Nov 02 01:14:17.157461 ip-10-0-137-1 systemd[1]: Starting Kubernetes Kubelet... Nov 02 01:14:17.555377 ip-10-0-137-1 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:17.555377 ip-10-0-137-1 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 01:14:17.555377 ip-10-0-137-1 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:17.555377 ip-10-0-137-1 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:17.561869 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561752 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.561869 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561868 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.561869 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561872 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561875 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561879 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561882 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561887 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561892 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561895 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561898 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561901 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561904 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561907 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561909 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561912 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561914 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561917 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561920 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561923 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561925 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561928 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.561961 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561931 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561933 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561936 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561938 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561941 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561943 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561946 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561949 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561951 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561954 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561956 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561959 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561963 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561966 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561969 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561972 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561975 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561978 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561980 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.562435 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561983 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561985 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561988 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561990 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561993 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561995 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.561998 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562000 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562003 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562006 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562009 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562011 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562013 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562016 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562018 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562022 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562025 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562027 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562030 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562033 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.562978 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562035 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562038 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562041 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562043 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562046 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562049 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562052 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562054 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562057 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562060 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562062 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562065 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562068 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562070 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562073 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562075 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562078 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562080 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562083 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562086 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.563469 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562088 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562091 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562093 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562096 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562098 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562101 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562104 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562130 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562134 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562137 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562140 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562143 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562148 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562151 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562154 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562157 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562160 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562163 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.563979 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562166 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562169 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562171 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562174 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562177 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562179 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562183 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.562993 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563001 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563003 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563006 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563009 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563012 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563015 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563018 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563020 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563023 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563026 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563029 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563031 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.564446 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563038 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563041 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563044 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563047 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563050 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563052 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563055 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563058 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563061 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563063 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563066 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563069 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563071 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563075 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563078 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563080 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563083 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563086 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563089 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563091 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.564918 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563094 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563097 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563101 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563105 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563107 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563110 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563128 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563133 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563136 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563139 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563142 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563145 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563150 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563153 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563156 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563158 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563161 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563164 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563167 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.565402 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563169 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563172 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563175 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563177 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563180 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563183 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563185 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563188 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563191 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563194 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563196 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563199 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563202 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563205 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563208 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563211 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563214 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563217 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563220 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563222 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.565856 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563225 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563227 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563230 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563233 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563236 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563240 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563243 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563246 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563248 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563251 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563254 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563256 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563259 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563261 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563264 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563267 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563270 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563272 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563275 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.566338 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563277 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563280 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563283 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563286 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563289 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563292 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563294 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563297 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563299 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563302 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563305 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563307 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563310 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563312 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563315 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563866 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563871 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563874 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563881 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.566784 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563885 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563888 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563891 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563894 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563897 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563899 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563902 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563904 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563907 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563910 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563912 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563915 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563917 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563920 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563928 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563932 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563934 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563937 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563940 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563942 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.567242 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563945 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563948 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563950 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563953 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563955 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563958 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563960 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563963 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563966 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563968 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563971 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563975 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563978 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563980 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563983 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563986 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563989 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563991 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563994 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.567722 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563996 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.563999 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564001 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564004 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564006 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564009 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564011 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564014 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564022 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564025 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564028 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564030 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564033 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564036 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564038 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564041 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564044 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564046 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564049 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564051 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.568181 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564054 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564056 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564059 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564062 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564065 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564067 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564070 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564072 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564075 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564077 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564080 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564082 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564085 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564087 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564089 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564092 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564094 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564097 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564099 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564102 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.568653 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564104 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564125 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564129 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564131 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564134 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564137 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564140 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564143 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564145 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564148 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564150 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564159 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564161 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564168 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564171 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564174 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564177 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564179 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564182 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.569166 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564185 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564187 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564190 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.564193 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565251 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565273 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565284 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565290 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565294 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565298 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565302 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565306 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565310 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565313 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565318 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565327 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565331 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565334 2573 flags.go:64] FLAG: --cgroup-root="" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565337 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565340 2573 flags.go:64] FLAG: --client-ca-file="" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565343 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565346 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565353 2573 flags.go:64] FLAG: --cluster-domain="" Nov 02 01:14:17.569602 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565356 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565359 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565363 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565369 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565373 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565376 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565379 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565382 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565385 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565389 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565392 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565395 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565398 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565403 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565406 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565409 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565413 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565416 2573 flags.go:64] FLAG: --enable-server="true" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565419 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565426 2573 flags.go:64] FLAG: --event-burst="100" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565429 2573 flags.go:64] FLAG: --event-qps="50" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565432 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565435 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565438 2573 flags.go:64] FLAG: --eviction-hard="" Nov 02 01:14:17.570172 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565442 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565451 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565454 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565457 2573 flags.go:64] FLAG: --eviction-soft="" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565461 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565464 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565466 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565469 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565472 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565475 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565478 2573 flags.go:64] FLAG: --feature-gates="" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565484 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565487 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565490 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565493 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565496 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565499 2573 flags.go:64] FLAG: --help="false" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565502 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565505 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565508 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565511 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565514 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565518 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 01:14:17.570759 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565520 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565524 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565526 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565529 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565532 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565535 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565538 2573 flags.go:64] FLAG: --kube-reserved="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565541 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565544 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565547 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565550 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565558 2573 flags.go:64] FLAG: --lock-file="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565561 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565564 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565567 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565573 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565575 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565578 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565581 2573 flags.go:64] FLAG: --logging-format="text" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565584 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565590 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565593 2573 flags.go:64] FLAG: --manifest-url="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565596 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565601 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565604 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 01:14:17.571314 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565608 2573 flags.go:64] FLAG: --max-pods="110" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565611 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565614 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565616 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565620 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565623 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565626 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565629 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565636 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565639 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565642 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565649 2573 flags.go:64] FLAG: --pod-cidr="" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565651 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565655 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565658 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565661 2573 flags.go:64] FLAG: --port="10250" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565664 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565667 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-025a43f96557ca686" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565670 2573 flags.go:64] FLAG: --qos-reserved="" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565678 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565682 2573 flags.go:64] FLAG: --register-node="true" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565685 2573 flags.go:64] FLAG: --register-with-taints="" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565688 2573 flags.go:64] FLAG: --registry-burst="10" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565691 2573 flags.go:64] FLAG: --registry-qps="5" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565694 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565697 2573 flags.go:64] FLAG: --reserved-memory="" Nov 02 01:14:17.571904 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565701 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565706 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565709 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565712 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565715 2573 flags.go:64] FLAG: --runonce="false" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565718 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565721 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565724 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565727 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565730 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565733 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565736 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565739 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565742 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565745 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565748 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565751 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565754 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565757 2573 flags.go:64] FLAG: --system-cgroups="" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565759 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565765 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565768 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565770 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565777 2573 flags.go:64] FLAG: --tls-min-version="" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565785 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 01:14:17.572538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565788 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565800 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565803 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565806 2573 flags.go:64] FLAG: --v="2" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565811 2573 flags.go:64] FLAG: --version="false" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565815 2573 flags.go:64] FLAG: --vmodule="" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565819 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.565822 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566013 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566017 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566019 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566022 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566025 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566027 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566030 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566033 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566035 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566039 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566041 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566044 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566046 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.573096 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566049 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566052 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566054 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566057 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566060 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566062 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566065 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566069 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566072 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566075 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566078 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566081 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566084 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566092 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566095 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566098 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566100 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566103 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566105 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.573644 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566110 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566126 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566129 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566131 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566134 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566137 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566139 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566142 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566145 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566147 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566150 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566153 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566155 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566158 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566161 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566163 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566166 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566168 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566171 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566173 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.574240 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566176 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566179 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566181 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566184 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566186 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566189 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566191 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566199 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566202 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566205 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566207 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566210 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566214 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566217 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566220 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566222 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566225 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566228 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566230 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566233 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.575047 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566235 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566238 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566240 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566243 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566246 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566248 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566251 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566253 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566256 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566258 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566261 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566263 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566266 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566268 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566271 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566274 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566277 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566279 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566282 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566284 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.575586 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566294 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566297 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566300 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566305 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566311 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566314 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566317 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566320 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566322 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566325 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566327 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566330 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566332 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566335 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.566362 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:17.576054 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566406 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566411 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566414 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566417 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566420 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566423 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566425 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566429 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566433 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566436 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566439 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566441 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566444 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566447 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566449 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566452 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566454 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566457 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566465 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.576440 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566467 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566470 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566474 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566478 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566481 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566483 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566486 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566488 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566491 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566494 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566502 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566505 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566508 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566510 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566513 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566515 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566518 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566521 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566523 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566526 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.576885 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566534 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566537 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566540 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566542 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566545 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566548 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566550 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566553 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566555 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566558 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566560 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566563 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566571 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566574 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566576 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566580 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566583 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566585 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566588 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566590 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.577370 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566593 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566595 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566598 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566600 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566603 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566605 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566608 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566610 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566613 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566616 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566618 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566621 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566623 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566625 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566628 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566630 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566633 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566636 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566638 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566641 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.577825 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566643 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566646 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566650 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566654 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566657 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566667 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566671 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566675 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566678 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566680 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566683 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566686 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566688 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566691 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566694 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566696 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566699 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566701 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566704 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.578324 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566706 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566709 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566712 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566714 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566718 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566720 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566723 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.566725 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.576643 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.576661 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.576686 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.576691 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576767 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576774 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576778 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576781 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576784 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576786 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.578754 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576789 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576792 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576795 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576798 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576801 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576803 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576806 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576809 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576811 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576814 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576816 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576819 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576822 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576824 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576827 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576830 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576833 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576835 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576838 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.579186 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576841 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576844 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576847 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576851 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576856 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576860 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576864 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576867 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576870 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576872 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576875 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576877 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576880 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576882 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576885 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576888 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576891 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576894 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576897 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.579669 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576899 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576902 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576905 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576908 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576911 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576913 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576916 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576918 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576921 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576923 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576926 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576928 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576931 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576934 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576936 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576939 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576941 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576944 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576947 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.580310 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576950 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576952 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576955 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576958 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576960 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576962 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576965 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576967 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576970 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576973 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576975 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576977 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576980 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576983 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576985 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576987 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576990 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576993 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576995 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.576998 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.580774 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577001 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577003 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577006 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577008 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577010 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577013 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577016 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577019 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577021 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577024 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577026 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577029 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577031 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577034 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577037 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577040 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577042 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577045 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577047 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.581256 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577050 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577052 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577055 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577057 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.577096 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577264 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577270 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577273 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577276 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577279 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577284 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577288 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577291 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577293 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577296 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577299 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.581689 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577301 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577304 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577306 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577309 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577312 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577315 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577320 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577323 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577326 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577328 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577331 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577334 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577336 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577339 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577341 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577344 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577347 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577350 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577352 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.582058 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577355 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577358 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577360 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577363 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577365 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577368 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577370 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577374 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577377 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577379 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577382 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577385 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577388 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577390 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577393 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577396 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577398 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577401 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577403 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.582555 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577406 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577409 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577412 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577414 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577417 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577419 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577422 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577424 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577427 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577429 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577432 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577434 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577437 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577439 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577442 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577444 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577447 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577449 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577452 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577454 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.583041 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577457 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577459 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577462 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577465 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577467 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577470 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577472 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577475 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577478 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577480 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577483 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577486 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577490 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577494 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577497 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577500 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577503 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577506 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577509 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.583528 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577511 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577514 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577517 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577519 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577522 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577524 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577527 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577529 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577532 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577534 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577537 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577539 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577542 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577544 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577547 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577550 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.584035 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577553 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.577575 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577608 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577612 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577616 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577619 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577622 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577626 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577629 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577632 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577636 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577639 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577643 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577645 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577648 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577651 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:17.584538 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577654 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577656 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577659 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577661 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577664 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577666 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577669 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577672 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577674 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577676 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577679 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577681 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577684 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577687 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577689 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577691 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577695 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577698 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577700 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577703 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:17.584929 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577706 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577709 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577711 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577714 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577717 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577720 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577722 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577725 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577727 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577730 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577732 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577735 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577737 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577740 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577742 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577745 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577748 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577750 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577753 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:17.585422 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577755 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577758 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577760 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577763 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577766 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577768 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577771 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577773 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577775 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577778 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577781 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577783 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577786 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577789 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577791 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577794 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577796 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577799 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577802 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:17.585864 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577804 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577807 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577809 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577812 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577815 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577817 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577820 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577822 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577825 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577827 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577830 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577833 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577836 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577838 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577841 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577844 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577847 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577850 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577853 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:17.586360 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577856 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577858 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577861 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577865 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577867 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577871 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577873 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577878 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577881 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577883 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577886 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577892 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577895 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577898 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:17.577900 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.577975 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.580770 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.581715 2573 server.go:1014] "Starting client certificate rotation" Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.581815 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:14:17.586862 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.581847 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:14:17.587560 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.583611 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:17.587560 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.586779 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:17.598272 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.598254 2573 log.go:25] "Validated CRI v1 runtime API" Nov 02 01:14:17.601724 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.601710 2573 log.go:25] "Validated CRI v1 image API" Nov 02 01:14:17.602815 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.602788 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 01:14:17.607098 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.607079 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a1c5980-4a96-4861-ac74-e39d4d89d2ac:/dev/nvme0n1p3 a4741b66-d346-4584-a51f-21ad1107403a:/dev/nvme0n1p4] Nov 02 01:14:17.607153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.607098 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 01:14:17.614194 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.613938 2573 manager.go:217] Machine: {Timestamp:2026-11-02 01:14:17.611901282 +0000 UTC m=+0.340218203 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100046 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c4a499b421177b16f38fb93bb757 SystemUUID:ec22c4a4-99b4-2117-7b16-f38fb93bb757 BootID:60a04206-17f5-43b3-9423-b229ebad30eb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:9d:33:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:9d:33:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:6b:29:8b:c2:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 01:14:17.614194 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.614183 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 01:14:17.614326 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.614311 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 01:14:17.617584 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617555 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 01:14:17.617756 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617584 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-1.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 01:14:17.617756 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617757 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 01:14:17.617885 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617765 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 01:14:17.617885 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617803 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:17.617885 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617823 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:17.617885 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.617845 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 01:14:17.618963 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.618953 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 01:14:17.623051 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623039 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 01:14:17.623178 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623168 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 01:14:17.623215 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623200 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 02 01:14:17.623215 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623211 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 01:14:17.623267 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623225 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 01:14:17.623267 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623234 2573 kubelet.go:405] "Adding apiserver pod source" Nov 02 01:14:17.623267 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.623242 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 01:14:17.624331 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.624317 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:17.627449 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.627430 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 01:14:17.634315 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.634299 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:17.635101 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635090 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 01:14:17.635171 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635135 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 01:14:17.635284 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635273 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 01:14:17.635315 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635295 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 01:14:17.635315 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635303 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 01:14:17.635315 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635312 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 01:14:17.635405 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635322 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 01:14:17.635405 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635331 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 01:14:17.635985 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635975 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 01:14:17.636014 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635986 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 01:14:17.636014 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.635993 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 01:14:17.636014 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.636000 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 01:14:17.636014 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.636008 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 01:14:17.636160 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.636017 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 01:14:17.636651 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.636642 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 01:14:17.636679 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.636652 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 01:14:17.640323 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.640279 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-1.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 01:14:17.640496 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.640478 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67pl7" Nov 02 01:14:17.640659 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.640646 2573 server.go:1263] "Started kubelet" Nov 02 01:14:17.640909 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.640843 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 01:14:17.640977 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.640930 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 01:14:17.641375 ip-10-0-137-1 systemd[1]: Started Kubernetes Kubelet. Nov 02 01:14:17.641642 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.641601 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 01:14:17.642548 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.642535 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 01:14:17.643675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.643647 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 02 01:14:17.646179 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.646155 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-1.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 01:14:17.646269 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.646198 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 01:14:17.647319 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.647298 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 01:14:17.648961 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.648945 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 01:14:17.650337 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.650321 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 01:14:17.650467 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.650424 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:17.651020 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651005 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 01:14:17.651144 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651134 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 01:14:17.651273 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651004 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 01:14:17.651273 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651187 2573 factory.go:55] Registering systemd factory Nov 02 01:14:17.651458 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651302 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 02 01:14:17.651458 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651314 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 02 01:14:17.651458 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651305 2573 factory.go:223] Registration of the systemd container factory successfully Nov 02 01:14:17.651922 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.651903 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67pl7" Nov 02 01:14:17.652095 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.652083 2573 factory.go:153] Registering CRI-O factory Nov 02 01:14:17.652155 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.652101 2573 factory.go:223] Registration of the crio container factory successfully Nov 02 01:14:17.652198 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.652186 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 01:14:17.652227 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.652211 2573 factory.go:103] Registering Raw factory Nov 02 01:14:17.652227 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.652226 2573 manager.go:1199] Started watching for new ooms in manager Nov 02 01:14:17.653456 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.653422 2573 manager.go:319] Starting recovery of all containers Nov 02 01:14:17.653626 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.653601 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-1.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 01:14:17.655006 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.653879 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-1.us-east-2.compute.internal.18e41560ed2eec3f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-1.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-1.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:17.640619071 +0000 UTC m=+0.368935991,LastTimestamp:2026-11-02 01:14:17.640619071 +0000 UTC m=+0.368935991,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-1.us-east-2.compute.internal,}" Nov 02 01:14:17.659975 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.659946 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 01:14:17.666299 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.666280 2573 manager.go:324] Recovery completed Nov 02 01:14:17.667524 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.667452 2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 01:14:17.670245 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.670231 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.672111 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672093 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.672193 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672143 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.672193 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672155 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.672669 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672657 2573 cpu_manager.go:226] "Starting" policy="none" Nov 02 01:14:17.672669 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672668 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.672753 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672669 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 01:14:17.672753 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.672706 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 01:14:17.673400 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.673386 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.673458 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.673412 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.673458 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.673423 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.674691 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.674681 2573 policy_none.go:50] "Start" Nov 02 01:14:17.674723 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.674696 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 01:14:17.674723 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.674705 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 01:14:17.676590 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.676578 2573 policy_none.go:44] "Start" Nov 02 01:14:17.721795 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.720799 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.724813 2573 manager.go:341] "Starting Device Plugin manager" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.724838 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.724847 2573 server.go:87] "Starting device plugin registration server" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.725157 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.725172 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.725264 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.725344 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.725353 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.725881 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.725931 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.737774 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.739106 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.739169 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.739190 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.739228 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 01:14:17.743450 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.743021 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:17.826183 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.826074 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.827102 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.827085 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.827204 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.827132 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.827204 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.827146 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.827204 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.827176 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.836309 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.836289 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.836393 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.836313 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-1.us-east-2.compute.internal\": node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:17.839607 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.839589 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal"] Nov 02 01:14:17.839655 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.839645 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.840472 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.840457 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.840521 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.840484 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.840521 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.840494 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.842705 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.842693 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.842842 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.842829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.842876 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.842860 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.843707 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.843682 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.843788 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.843710 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.843788 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.843724 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.843788 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.843736 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.843788 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.843753 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.843788 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.843762 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.845903 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.845885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.845966 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.845921 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:17.846594 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.846579 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:17.846649 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.846615 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:17.846649 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.846630 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:17.853105 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.853089 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:17.856267 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.856252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0100e2035cafa00853427683c7ebd01f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"0100e2035cafa00853427683c7ebd01f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.856341 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.856282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.856341 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.856308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.876802 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.876785 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.879362 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.879345 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.953432 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:17.953405 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:17.956463 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.956439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0100e2035cafa00853427683c7ebd01f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"0100e2035cafa00853427683c7ebd01f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.956547 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.956488 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0100e2035cafa00853427683c7ebd01f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"0100e2035cafa00853427683c7ebd01f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.956547 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.956532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.956623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.956554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.956623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.956606 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:17.956623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:17.956607 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:18.054528 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.054479 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.155382 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.155314 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.183631 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.183609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:18.185781 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.185764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:18.255600 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.255567 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.356182 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.356149 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.456664 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.456596 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.546815 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.546785 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:18.557086 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.557064 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.581386 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.581368 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 01:14:18.581505 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.581489 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 01:14:18.581542 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.581517 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 01:14:18.657944 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.657916 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.662054 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.662019 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:09:18 +0000 UTC" deadline="2028-04-01 23:27:13.272593598 +0000 UTC" Nov 02 01:14:18.662054 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.662051 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12406h12m54.610545243s" Nov 02 01:14:18.758856 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.758827 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.857145 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:18.857097 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990ecd6a432f4311162cd85719f32ab0.slice/crio-fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64 WatchSource:0}: Error finding container fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64: Status 404 returned error can't find the container with id fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64 Nov 02 01:14:18.857411 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:18.857395 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0100e2035cafa00853427683c7ebd01f.slice/crio-2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b WatchSource:0}: Error finding container 2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b: Status 404 returned error can't find the container with id 2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b Nov 02 01:14:18.859567 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.859553 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:18.862604 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.862590 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:14:18.880317 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:18.880300 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:18.960642 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:18.960590 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-east-2.compute.internal\" not found" Nov 02 01:14:19.021938 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.021862 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:19.050980 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.050947 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:19.064170 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.064146 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:19.065497 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.065485 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:19.075123 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.075104 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:19.502057 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.502023 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:19.624100 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.624010 2573 apiserver.go:52] "Watching apiserver" Nov 02 01:14:19.635058 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.634228 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 01:14:19.635058 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.634614 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal","openshift-dns/node-resolver-vts4p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-mcfgw","kube-system/konnectivity-agent-qjmzq","openshift-image-registry/node-ca-zttdn","openshift-multus/multus-j5k6h","openshift-network-operator/iptables-alerter-x82sf","openshift-network-diagnostics/network-check-target-4x747","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5","openshift-cluster-node-tuning-operator/tuned-kgnc2","openshift-multus/multus-additional-cni-plugins-8m9jp","openshift-multus/network-metrics-daemon-wx6pd","kube-system/global-pull-secret-syncer-tqwk8"] Nov 02 01:14:19.637763 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.637730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.640103 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.640073 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:19.640250 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.640196 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:19.642498 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.642467 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-87cgr\"" Nov 02 01:14:19.642623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.642473 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 01:14:19.642623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.642504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.642623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.642557 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 01:14:19.642623 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.642576 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:19.642623 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.642586 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.642865 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.642651 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.644996 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.644974 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.647443 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.647425 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.650034 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.650012 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 01:14:19.650391 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.650369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.653518 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.653497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:19.653611 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.653571 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:19.656084 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.656061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.658285 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.658268 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.661497 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.661472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.665940 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.665292 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.666048 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.665998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-hostroot\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666110 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-multus-certs\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666110 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-ovn\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.666250 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-slash\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.666250 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-systemd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.666250 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-netd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.666397 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-serviceca\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.666397 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-cnibin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666397 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-cni-binary-copy\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666397 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-log-socket\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.666397 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.666647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-socket-dir-parent\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz8qd\" (UniqueName: \"kubernetes.io/projected/55f40d17-3c79-4d12-9f72-58c903869221-kube-api-access-cz8qd\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:19.666647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-kubelet-config\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.666647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-system-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-bin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666842 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-multus\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666842 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-conf-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666842 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-etc-kubernetes\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.666842 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.666825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-etc-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667211 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-bin\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667211 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-config\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667211 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-agent-certs\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.667211 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xx9j\" (UniqueName: \"kubernetes.io/projected/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-kube-api-access-8xx9j\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.667415 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-dbus\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.667415 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667270 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:19.667415 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-k8s-cni-cncf-io\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.667415 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-kubelet\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.667415 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-daemon-config\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.667415 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd8x\" (UniqueName: \"kubernetes.io/projected/6cfdec63-7004-4a85-91dd-6a7a6db01761-kube-api-access-zsd8x\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-netns\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-node-log\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72cd\" (UniqueName: \"kubernetes.io/projected/4809d979-ef70-435c-b1e5-e403fa49988d-kube-api-access-l72cd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-konnectivity-ca\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-host\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.667686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-kubelet\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-var-lib-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4809d979-ef70-435c-b1e5-e403fa49988d-ovn-node-metrics-cert\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-netns\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.667889 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-systemd-units\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.668130 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-env-overrides\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.668130 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.667951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-script-lib\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.669006 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.668981 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 01:14:19.671587 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.671563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.693778 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.693748 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8dcd9" Nov 02 01:14:19.708730 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.708704 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8dcd9" Nov 02 01:14:19.744526 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.744470 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"0100e2035cafa00853427683c7ebd01f","Type":"ContainerStarted","Data":"2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b"} Nov 02 01:14:19.745844 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.745816 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"990ecd6a432f4311162cd85719f32ab0","Type":"ContainerStarted","Data":"fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64"} Nov 02 01:14:19.752377 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.752337 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 01:14:19.752470 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.752426 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 01:14:19.753515 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.753485 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.753733 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.753694 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.753826 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.753796 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754076 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754089 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754156 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8rk5x\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754234 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754283 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85fmt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754289 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlhpz\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754361 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754403 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754452 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754487 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5kn9m\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754571 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754588 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75lmf\"" Nov 02 01:14:19.754678 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.754659 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 01:14:19.755639 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755617 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pl9l\"" Nov 02 01:14:19.755748 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755652 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 01:14:19.755748 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755654 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 01:14:19.755748 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755742 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.755952 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755855 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 01:14:19.755952 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755857 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 01:14:19.755952 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755861 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 01:14:19.755952 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755863 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67z2p\"" Nov 02 01:14:19.755952 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755904 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 01:14:19.755952 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.755920 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 01:14:19.756326 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.756057 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 01:14:19.757662 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.757639 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 01:14:19.757863 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.757839 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 01:14:19.757947 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.757900 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tmdq2\"" Nov 02 01:14:19.769255 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.769219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:19.769373 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.769260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5aaf1d9-2a69-4a0f-9503-2217191d7131-host-slash\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.769373 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.769287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cnibin\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.769583 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.769310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.769711 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.769672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv252\" (UniqueName: \"kubernetes.io/projected/15670b0f-869e-49bf-9da0-9bf6ea903f2c-kube-api-access-mv252\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-daemon-config\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-netns\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72cd\" (UniqueName: \"kubernetes.io/projected/4809d979-ef70-435c-b1e5-e403fa49988d-kube-api-access-l72cd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-host\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-netns\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-sys-fs\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.771999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rdt\" (UniqueName: \"kubernetes.io/projected/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-kube-api-access-f7rdt\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-kubernetes\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-script-lib\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-hosts-file\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-host\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-ovn\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-slash\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772262 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-daemon-config\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.772927 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-cnibin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772317 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-cni-binary-copy\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-socket-dir-parent\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772403 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-netns\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-tmp-dir\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772419 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-host\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772496 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-ovn\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772509 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-netns\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772551 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-cnibin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-conf\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-slash\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-sys\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.772965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz8qd\" (UniqueName: \"kubernetes.io/projected/55f40d17-3c79-4d12-9f72-58c903869221-kube-api-access-cz8qd\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-kubelet-config\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-bin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.773792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-conf-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-cni-binary-copy\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-etc-kubernetes\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-agent-certs\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773304 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-socket-dir-parent\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773328 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-script-lib\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773356 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-bin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773406 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-conf-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xx9j\" (UniqueName: \"kubernetes.io/projected/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-kube-api-access-8xx9j\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-dbus\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-etc-kubernetes\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773496 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-kubelet-config\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-k8s-cni-cncf-io\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd8x\" (UniqueName: \"kubernetes.io/projected/6cfdec63-7004-4a85-91dd-6a7a6db01761-kube-api-access-zsd8x\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773580 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-dbus\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773739 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-k8s-cni-cncf-io\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-kubelet\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.774675 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-socket-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773800 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-kubelet\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-device-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773838 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-node-log\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.773937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-konnectivity-ca\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-registration-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-modprobe-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-systemd\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-run\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774207 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-tuned\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt9j5\" (UniqueName: \"kubernetes.io/projected/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-kube-api-access-dt9j5\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-kubelet\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-var-lib-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774378 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-konnectivity-ca\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.775543 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4809d979-ef70-435c-b1e5-e403fa49988d-ovn-node-metrics-cert\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774482 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rsb\" (UniqueName: \"kubernetes.io/projected/d5aaf1d9-2a69-4a0f-9503-2217191d7131-kube-api-access-46rsb\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.774655 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.774733 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:20.274712102 +0000 UTC m=+3.003029035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774769 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-var-lib-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-kubelet\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.774871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775049 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-node-log\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4lp\" (UniqueName: \"kubernetes.io/projected/779489ef-236e-4998-a739-964989ab5d07-kube-api-access-gk4lp\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-systemd-units\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-env-overrides\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-systemd-units\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-hostroot\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.775965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-multus-certs\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5aaf1d9-2a69-4a0f-9503-2217191d7131-iptables-alerter-script\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.776327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysconfig\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-lib-modules\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776178 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-hostroot\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776210 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-multus-certs\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-var-lib-kubelet\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-systemd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-netd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-systemd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776415 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-netd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-serviceca\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776508 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-env-overrides\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776597 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-tmp\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-log-socket\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776892 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-log-socket\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777153 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.776749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-serviceca\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.777021 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.777089 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret podName:dc2693cb-eac1-4cd0-b76c-db35732e9635 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:20.277060801 +0000 UTC m=+3.005377728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret") pod "global-pull-secret-syncer-tqwk8" (UID: "dc2693cb-eac1-4cd0-b76c-db35732e9635") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-system-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777224 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-system-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-multus\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777280 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-multus\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-etc-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-bin\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-config\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777473 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-etc-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.777974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.777633 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-bin\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.778621 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.778077 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-config\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.778621 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.778157 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4809d979-ef70-435c-b1e5-e403fa49988d-ovn-node-metrics-cert\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.778819 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.778794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-agent-certs\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.781846 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.781800 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:19.781846 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.781823 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:19.781846 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.781838 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fr7fg for pod openshift-network-diagnostics/network-check-target-4x747: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:19.782032 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:19.781891 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg podName:f6ca2c0e-a09c-4c72-bb67-c595dad9050c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:20.28187465 +0000 UTC m=+3.010191560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fr7fg" (UniqueName: "kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg") pod "network-check-target-4x747" (UID: "f6ca2c0e-a09c-4c72-bb67-c595dad9050c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:19.788394 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.788348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72cd\" (UniqueName: \"kubernetes.io/projected/4809d979-ef70-435c-b1e5-e403fa49988d-kube-api-access-l72cd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.788707 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.788683 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xx9j\" (UniqueName: \"kubernetes.io/projected/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-kube-api-access-8xx9j\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.789003 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.788982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd8x\" (UniqueName: \"kubernetes.io/projected/6cfdec63-7004-4a85-91dd-6a7a6db01761-kube-api-access-zsd8x\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.789331 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.789310 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz8qd\" (UniqueName: \"kubernetes.io/projected/55f40d17-3c79-4d12-9f72-58c903869221-kube-api-access-cz8qd\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:19.878630 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-tuned\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.878630 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9j5\" (UniqueName: \"kubernetes.io/projected/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-kube-api-access-dt9j5\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.878630 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rsb\" (UniqueName: \"kubernetes.io/projected/d5aaf1d9-2a69-4a0f-9503-2217191d7131-kube-api-access-46rsb\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4lp\" (UniqueName: \"kubernetes.io/projected/779489ef-236e-4998-a739-964989ab5d07-kube-api-access-gk4lp\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5aaf1d9-2a69-4a0f-9503-2217191d7131-iptables-alerter-script\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysconfig\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-lib-modules\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-var-lib-kubelet\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-tmp\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.878891 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5aaf1d9-2a69-4a0f-9503-2217191d7131-host-slash\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.878981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cnibin\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv252\" (UniqueName: \"kubernetes.io/projected/15670b0f-869e-49bf-9da0-9bf6ea903f2c-kube-api-access-mv252\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-sys-fs\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rdt\" (UniqueName: \"kubernetes.io/projected/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-kube-api-access-f7rdt\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-kubernetes\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879191 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879278 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.879399 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879351 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.879924 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879409 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879924 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879475 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cnibin\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.879924 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-hosts-file\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.879924 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879738 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysconfig\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.879924 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879761 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-lib-modules\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880186 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880038 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-var-lib-kubelet\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880186 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880101 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.880186 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880114 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.880186 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880192 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5aaf1d9-2a69-4a0f-9503-2217191d7131-host-slash\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.879195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-hosts-file\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880181 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-kubernetes\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-host\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880278 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-tmp-dir\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-conf\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-sys\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880423 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-socket-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-device-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-registration-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-modprobe-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-systemd\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880541 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-sys-fs\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-run\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880626 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-run\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880638 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880683 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-host\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-socket-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880746 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5aaf1d9-2a69-4a0f-9503-2217191d7131-iptables-alerter-script\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880779 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-device-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880825 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-registration-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.880869 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880831 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-sys\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.881405 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-modprobe-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.881405 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880940 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-tmp-dir\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.881405 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-conf\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.881405 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.880964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-systemd\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.881651 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.881628 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.883276 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.883251 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-tuned\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.883446 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.883354 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-tmp\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.892506 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.892478 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4lp\" (UniqueName: \"kubernetes.io/projected/779489ef-236e-4998-a739-964989ab5d07-kube-api-access-gk4lp\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:19.892847 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.892824 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rsb\" (UniqueName: \"kubernetes.io/projected/d5aaf1d9-2a69-4a0f-9503-2217191d7131-kube-api-access-46rsb\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:19.893522 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.893466 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rdt\" (UniqueName: \"kubernetes.io/projected/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-kube-api-access-f7rdt\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:19.893522 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.893492 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9j5\" (UniqueName: \"kubernetes.io/projected/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-kube-api-access-dt9j5\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:19.894825 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.894804 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv252\" (UniqueName: \"kubernetes.io/projected/15670b0f-869e-49bf-9da0-9bf6ea903f2c-kube-api-access-mv252\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:19.953235 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.953199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5k6h" Nov 02 01:14:19.983186 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.983162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:19.986704 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.986681 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:14:19.989373 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:19.989349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:19.995345 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:19.995318 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cdfa2e_154d_443b_b5ef_7fbea8c1b3fa.slice/crio-3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c WatchSource:0}: Error finding container 3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c: Status 404 returned error can't find the container with id 3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c Nov 02 01:14:20.001273 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.001252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:14:20.002109 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.002080 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040c33cc_4366_4e4a_8d93_3f35a1bd5cb8.slice/crio-f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595 WatchSource:0}: Error finding container f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595: Status 404 returned error can't find the container with id f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595 Nov 02 01:14:20.003184 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.003147 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4809d979_ef70_435c_b1e5_e403fa49988d.slice/crio-bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759 WatchSource:0}: Error finding container bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759: Status 404 returned error can't find the container with id bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759 Nov 02 01:14:20.009363 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.009344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vts4p" Nov 02 01:14:20.012477 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.012455 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aaf1d9_2a69_4a0f_9503_2217191d7131.slice/crio-bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f WatchSource:0}: Error finding container bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f: Status 404 returned error can't find the container with id bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f Nov 02 01:14:20.019346 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.019320 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d617c8_7375_4b58_93b2_7eb39c4bb8c0.slice/crio-c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea WatchSource:0}: Error finding container c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea: Status 404 returned error can't find the container with id c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea Nov 02 01:14:20.023707 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.023692 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:14:20.028382 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.028358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:14:20.038267 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.038237 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779489ef_236e_4998_a739_964989ab5d07.slice/crio-1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18 WatchSource:0}: Error finding container 1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18: Status 404 returned error can't find the container with id 1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18 Nov 02 01:14:20.038802 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.038752 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb712b5a_44f2_4a7b_b3da_7bf1e9253da4.slice/crio-f1bf9598da14a87043e68c32e327c38d46454099b86fbd4420c66a33f890586c WatchSource:0}: Error finding container f1bf9598da14a87043e68c32e327c38d46454099b86fbd4420c66a33f890586c: Status 404 returned error can't find the container with id f1bf9598da14a87043e68c32e327c38d46454099b86fbd4420c66a33f890586c Nov 02 01:14:20.047792 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.047774 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:14:20.062331 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:20.062295 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15670b0f_869e_49bf_9da0_9bf6ea903f2c.slice/crio-d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe WatchSource:0}: Error finding container d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe: Status 404 returned error can't find the container with id d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe Nov 02 01:14:20.283236 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.283118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:20.283236 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.283204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.283246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283317 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283369 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283388 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283398 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:21.283374501 +0000 UTC m=+4.011691433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283403 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:20.283470 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283400 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fr7fg for pod openshift-network-diagnostics/network-check-target-4x747: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:20.283760 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283468 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret podName:dc2693cb-eac1-4cd0-b76c-db35732e9635 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:21.283444507 +0000 UTC m=+4.011761415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret") pod "global-pull-secret-syncer-tqwk8" (UID: "dc2693cb-eac1-4cd0-b76c-db35732e9635") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:20.283760 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:20.283556 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg podName:f6ca2c0e-a09c-4c72-bb67-c595dad9050c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:21.283538383 +0000 UTC m=+4.011855305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fr7fg" (UniqueName: "kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg") pod "network-check-target-4x747" (UID: "f6ca2c0e-a09c-4c72-bb67-c595dad9050c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:20.712407 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.712202 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:20 +0000 UTC" deadline="2028-06-21 17:45:20.664200045 +0000 UTC" Nov 02 01:14:20.712407 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.712243 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14344h30m59.951961738s" Nov 02 01:14:20.775060 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.774921 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5k6h" event={"ID":"6cfdec63-7004-4a85-91dd-6a7a6db01761","Type":"ContainerStarted","Data":"e0e4f79ddc8a675e4ceede6fd99a69b7f24362c40ca650fd1cfd34adb614521e"} Nov 02 01:14:20.778601 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.778433 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerStarted","Data":"d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe"} Nov 02 01:14:20.786133 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.786049 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" event={"ID":"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4","Type":"ContainerStarted","Data":"f1bf9598da14a87043e68c32e327c38d46454099b86fbd4420c66a33f890586c"} Nov 02 01:14:20.788412 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.788381 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759"} Nov 02 01:14:20.804515 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.804464 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zttdn" event={"ID":"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8","Type":"ContainerStarted","Data":"f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595"} Nov 02 01:14:20.807879 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.807829 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" event={"ID":"779489ef-236e-4998-a739-964989ab5d07","Type":"ContainerStarted","Data":"1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18"} Nov 02 01:14:20.809981 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.809952 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vts4p" event={"ID":"30d617c8-7375-4b58-93b2-7eb39c4bb8c0","Type":"ContainerStarted","Data":"c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea"} Nov 02 01:14:20.824870 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.824739 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x82sf" event={"ID":"d5aaf1d9-2a69-4a0f-9503-2217191d7131","Type":"ContainerStarted","Data":"bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f"} Nov 02 01:14:20.830836 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:20.830657 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjmzq" event={"ID":"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa","Type":"ContainerStarted","Data":"3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c"} Nov 02 01:14:21.244454 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.244107 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:21.290412 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.290373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:21.290603 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.290442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:21.290603 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.290483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:21.290788 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290653 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:21.290788 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290675 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:21.290788 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290689 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fr7fg for pod openshift-network-diagnostics/network-check-target-4x747: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:21.290788 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290754 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg podName:f6ca2c0e-a09c-4c72-bb67-c595dad9050c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:23.290734239 +0000 UTC m=+6.019051166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fr7fg" (UniqueName: "kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg") pod "network-check-target-4x747" (UID: "f6ca2c0e-a09c-4c72-bb67-c595dad9050c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:21.291009 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290841 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:21.291009 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290880 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:23.290868044 +0000 UTC m=+6.019184967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:21.291009 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290941 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:21.291009 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.290976 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret podName:dc2693cb-eac1-4cd0-b76c-db35732e9635 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:23.290964773 +0000 UTC m=+6.019281695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret") pod "global-pull-secret-syncer-tqwk8" (UID: "dc2693cb-eac1-4cd0-b76c-db35732e9635") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:21.712806 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.712691 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:20 +0000 UTC" deadline="2028-07-03 13:08:39.834955085 +0000 UTC" Nov 02 01:14:21.712806 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.712755 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14627h54m18.122205455s" Nov 02 01:14:21.739741 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.739667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:21.739741 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.739684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:21.739952 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.739790 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:21.740395 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.740176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:21.740395 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.740300 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:21.740571 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:21.740425 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:21.863596 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.861815 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"0100e2035cafa00853427683c7ebd01f","Type":"ContainerStarted","Data":"9cc84f9aa0d1cca5f29f1026a36fd999db66e337581776dfdbcb479e3a26e56d"} Nov 02 01:14:21.878823 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:21.878255 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" podStartSLOduration=1.878236067 podStartE2EDuration="1.878236067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:21.878106707 +0000 UTC m=+4.606423639" watchObservedRunningTime="2026-11-02 01:14:21.878236067 +0000 UTC m=+4.606552997" Nov 02 01:14:22.876265 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:22.876231 2573 generic.go:356] "Generic (PLEG): container finished" podID="990ecd6a432f4311162cd85719f32ab0" containerID="6bd4ed71d5480a82bfc556b9487a005ecf88448cc9f53cacd90a1e571779d6f9" exitCode=0 Nov 02 01:14:22.877120 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:22.877096 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"990ecd6a432f4311162cd85719f32ab0","Type":"ContainerDied","Data":"6bd4ed71d5480a82bfc556b9487a005ecf88448cc9f53cacd90a1e571779d6f9"} Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.311775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.311847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.311879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312013 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312037 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312052 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fr7fg for pod openshift-network-diagnostics/network-check-target-4x747: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312114 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg podName:f6ca2c0e-a09c-4c72-bb67-c595dad9050c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:27.312094575 +0000 UTC m=+10.040411489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fr7fg" (UniqueName: "kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg") pod "network-check-target-4x747" (UID: "f6ca2c0e-a09c-4c72-bb67-c595dad9050c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312541 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312584 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:27.312571439 +0000 UTC m=+10.040888349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312649 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:23.312706 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.312675 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret podName:dc2693cb-eac1-4cd0-b76c-db35732e9635 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:27.312667387 +0000 UTC m=+10.040984298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret") pod "global-pull-secret-syncer-tqwk8" (UID: "dc2693cb-eac1-4cd0-b76c-db35732e9635") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:23.742316 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.742252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:23.742498 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.742395 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:23.742498 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.742476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:23.742647 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.742548 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:23.742647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.742607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:23.742770 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:23.742691 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:23.886964 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.886574 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"990ecd6a432f4311162cd85719f32ab0","Type":"ContainerStarted","Data":"dd2af87834ad561b3157d2336008243785304d2ab0bb832a003cd8ac7fef7d2b"} Nov 02 01:14:23.890076 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.890044 2573 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="0a8704a1d63aacb48c51b635da24e7222ca8b13431df3775e9a55c8b28f47d47" exitCode=0 Nov 02 01:14:23.890229 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.890093 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerDied","Data":"0a8704a1d63aacb48c51b635da24e7222ca8b13431df3775e9a55c8b28f47d47"} Nov 02 01:14:23.933269 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:23.933204 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" podStartSLOduration=3.933165548 podStartE2EDuration="3.933165548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:23.906539921 +0000 UTC m=+6.634856852" watchObservedRunningTime="2026-11-02 01:14:23.933165548 +0000 UTC m=+6.661482482" Nov 02 01:14:24.903925 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:24.903885 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qjmzq" event={"ID":"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa","Type":"ContainerStarted","Data":"17549d938d49a9edc3b0ae799c45515ac4ce80a30309d0602c6aa6c6595cc6a1"} Nov 02 01:14:25.741986 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:25.741951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:25.742191 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:25.742111 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:25.742542 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:25.742520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:25.742649 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:25.742627 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:25.742723 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:25.742683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:25.742792 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:25.742756 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:26.664030 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:26.663852 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:26.665363 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:26.665089 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:26.685038 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:26.684430 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qjmzq" podStartSLOduration=4.249191947 podStartE2EDuration="8.684412847s" totalImagesPullingTime="4.4352209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:24.924280144 +0000 UTC m=+7.652597083" watchObservedRunningTime="2026-11-02 01:14:26.684412847 +0000 UTC m=+9.412729770" Nov 02 01:14:26.922350 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:26.922199 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zttdn" event={"ID":"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8","Type":"ContainerStarted","Data":"45dfa26ae3b9a654574b1764713436d8d39a2887454a746df9400e35d7efc31d"} Nov 02 01:14:26.925775 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:26.925744 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" event={"ID":"779489ef-236e-4998-a739-964989ab5d07","Type":"ContainerStarted","Data":"b2fc7760231ff7e3aa39deebc15ce098e6df3cacbb29340174259e1e8a19aa0f"} Nov 02 01:14:26.940601 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:26.940542 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zttdn" podStartSLOduration=3.097778854 podStartE2EDuration="8.940522849s" totalImagesPullingTime="5.842743995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:26.94035005 +0000 UTC m=+9.668666984" watchObservedRunningTime="2026-11-02 01:14:26.940522849 +0000 UTC m=+9.668839783" Nov 02 01:14:27.350713 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.350678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:27.350864 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.350739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:27.350864 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.350768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:27.350944 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.350904 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:27.350944 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.350922 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:27.350944 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.350931 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fr7fg for pod openshift-network-diagnostics/network-check-target-4x747: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:27.351037 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.350986 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg podName:f6ca2c0e-a09c-4c72-bb67-c595dad9050c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:35.350973472 +0000 UTC m=+18.079290379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fr7fg" (UniqueName: "kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg") pod "network-check-target-4x747" (UID: "f6ca2c0e-a09c-4c72-bb67-c595dad9050c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:27.351091 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.351054 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:27.351146 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.351091 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:35.35107862 +0000 UTC m=+18.079395528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:27.351240 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.351152 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:27.351240 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.351211 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret podName:dc2693cb-eac1-4cd0-b76c-db35732e9635 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:35.351195969 +0000 UTC m=+18.079512878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret") pod "global-pull-secret-syncer-tqwk8" (UID: "dc2693cb-eac1-4cd0-b76c-db35732e9635") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:27.744038 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.744005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:27.744521 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.744138 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:27.745558 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.745534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:27.745951 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.745680 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:27.745951 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.745743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:27.745951 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:27.745820 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:27.928937 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:27.928905 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:14:28.464276 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.464235 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 01:14:28.737122 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.736315 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:14:28.464256884Z","UUID":"e6d706e7-4c4f-479a-9172-7dc025ab6a6d","Handler":null,"Name":"","Endpoint":""} Nov 02 01:14:28.738746 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.738460 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 01:14:28.738746 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.738493 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 01:14:28.937345 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.936976 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" event={"ID":"779489ef-236e-4998-a739-964989ab5d07","Type":"ContainerStarted","Data":"52b5507d64eb7d9ecf10a937be9a016260c00f1b8b35d75c013606ffb3aba4c0"} Nov 02 01:14:28.940063 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.940035 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vts4p" event={"ID":"30d617c8-7375-4b58-93b2-7eb39c4bb8c0","Type":"ContainerStarted","Data":"00fb280700094490f037d6566c50a39952c8678224164e7c58c1f3568f5e76cd"} Nov 02 01:14:28.960068 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:28.960017 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vts4p" podStartSLOduration=2.354698643 podStartE2EDuration="10.960005256s" totalImagesPullingTime="8.605306613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:28.959957463 +0000 UTC m=+11.688274394" watchObservedRunningTime="2026-11-02 01:14:28.960005256 +0000 UTC m=+11.688322185" Nov 02 01:14:29.740524 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.740491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:29.740714 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.740568 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:29.740714 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.740564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:29.740714 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:29.740689 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:29.741100 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:29.741018 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:29.741230 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:29.741101 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:29.945187 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.945117 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" event={"ID":"779489ef-236e-4998-a739-964989ab5d07","Type":"ContainerStarted","Data":"8350e8d985c40e3cc7eadd825924b9a65eac93554f619d01ddc175554a6ec42a"} Nov 02 01:14:29.947948 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.947908 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5k6h" event={"ID":"6cfdec63-7004-4a85-91dd-6a7a6db01761","Type":"ContainerStarted","Data":"e5225efb22fab6f3f2118ec3a3068143f1c97e5422e3a6ee60afdf8d45e16b68"} Nov 02 01:14:29.954813 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.954778 2573 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="55897de2bc81ead7334eaeca9437dcabd821daaeebc893f7262aea9e5304eb3c" exitCode=0 Nov 02 01:14:29.955392 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.955363 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerDied","Data":"55897de2bc81ead7334eaeca9437dcabd821daaeebc893f7262aea9e5304eb3c"} Nov 02 01:14:29.970803 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:29.970757 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" podStartSLOduration=2.894246903 podStartE2EDuration="11.970743338s" totalImagesPullingTime="9.076496435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:29.970471415 +0000 UTC m=+12.698788355" watchObservedRunningTime="2026-11-02 01:14:29.970743338 +0000 UTC m=+12.699060276" Nov 02 01:14:30.958584 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:30.958478 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x82sf" event={"ID":"d5aaf1d9-2a69-4a0f-9503-2217191d7131","Type":"ContainerStarted","Data":"7ca7ef2d11dcb952a16e7c86e2457325683d3cacfad7fd83164282e9c8d9f410"} Nov 02 01:14:30.977354 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:30.977300 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-j5k6h" podStartSLOduration=3.994605271 podStartE2EDuration="12.977283517s" totalImagesPullingTime="8.982678246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:30.019759164 +0000 UTC m=+12.748076097" watchObservedRunningTime="2026-11-02 01:14:30.977283517 +0000 UTC m=+13.705600448" Nov 02 01:14:30.977509 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:30.977412 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x82sf" podStartSLOduration=3.850638635 podStartE2EDuration="12.977407083s" totalImagesPullingTime="9.126768448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:30.976700575 +0000 UTC m=+13.705017511" watchObservedRunningTime="2026-11-02 01:14:30.977407083 +0000 UTC m=+13.705724015" Nov 02 01:14:31.673482 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.673453 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:31.673637 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.673622 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:14:31.674548 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.674524 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:14:31.740329 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.740227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:31.740536 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.740227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:31.740536 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:31.740386 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:31.740536 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.740240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:31.740536 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:31.740458 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:31.740752 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:31.740542 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:31.963874 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.963828 2573 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="6f0785c03a494ea2a78a3c20dcba0402543d023dcfe6f6d4523b039685b4462c" exitCode=0 Nov 02 01:14:31.964486 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.963940 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerDied","Data":"6f0785c03a494ea2a78a3c20dcba0402543d023dcfe6f6d4523b039685b4462c"} Nov 02 01:14:31.966898 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:31.966825 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" event={"ID":"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4","Type":"ContainerStarted","Data":"3dce12a55669ed6c4afaf63211c6bde53111e3009a4a100372863e73cfe96784"} Nov 02 01:14:32.020881 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:32.020761 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" podStartSLOduration=2.565914657 podStartE2EDuration="14.020740696s" totalImagesPullingTime="11.454826039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:32.02043759 +0000 UTC m=+14.748754522" watchObservedRunningTime="2026-11-02 01:14:32.020740696 +0000 UTC m=+14.749057628" Nov 02 01:14:33.740211 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:33.739943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:33.740664 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:33.740030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:33.740664 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:33.740287 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:33.740664 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:33.740094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:33.740664 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:33.740360 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:33.740664 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:33.740548 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:33.972931 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:33.972898 2573 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="947e42a90c65ab547f8a5a337a7d20a5d28892e627d7674725d37005b91b0ce8" exitCode=0 Nov 02 01:14:33.973103 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:33.972939 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerDied","Data":"947e42a90c65ab547f8a5a337a7d20a5d28892e627d7674725d37005b91b0ce8"} Nov 02 01:14:35.414851 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:35.414807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:35.414865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:35.414887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.414996 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415003 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415008 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415098 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415109 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fr7fg for pod openshift-network-diagnostics/network-check-target-4x747: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415077 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.415057862 +0000 UTC m=+34.143374773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415171 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret podName:dc2693cb-eac1-4cd0-b76c-db35732e9635 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.415160596 +0000 UTC m=+34.143477502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret") pod "global-pull-secret-syncer-tqwk8" (UID: "dc2693cb-eac1-4cd0-b76c-db35732e9635") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:35.415364 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.415182 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg podName:f6ca2c0e-a09c-4c72-bb67-c595dad9050c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.415176484 +0000 UTC m=+34.143493390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fr7fg" (UniqueName: "kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg") pod "network-check-target-4x747" (UID: "f6ca2c0e-a09c-4c72-bb67-c595dad9050c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:35.739647 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:35.739599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:35.739803 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:35.739667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:35.739803 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:35.739677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:35.739803 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.739756 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:35.740089 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.740051 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:35.740187 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:35.740143 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:36.982456 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.982271 2573 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="9019153bf44540db037e985b3d7014b066d92d675f14ca4bd47177dbaa4d8741" exitCode=0 Nov 02 01:14:36.982984 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.982351 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerDied","Data":"9019153bf44540db037e985b3d7014b066d92d675f14ca4bd47177dbaa4d8741"} Nov 02 01:14:36.985649 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.985632 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"9c16197ce156c0754e473f4e73cb1ed4216c4908e89f6c78ccd4c96248e63af0"} Nov 02 01:14:36.985718 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.985692 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"f10d9fbc2c623a931372f0826ae63c8e025c9f1a4dc7ca5ba91f6dafdda0953e"} Nov 02 01:14:36.985750 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.985723 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"b70c3e57837e2d63b4563da4a981fde585789f62b9483f56e94d9fc5048ac215"} Nov 02 01:14:36.985789 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.985757 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"837c27f160b96ff10eb5a72cdc3026f33c3af8f4fb082a84ac06dea189ee5298"} Nov 02 01:14:36.985789 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.985787 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"079532c58b047527cd943ede05a39b88b9274c8945cbb52627da9b99456235f4"} Nov 02 01:14:36.985850 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:36.985825 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"864bd68b26a5ac96c77a8551c10b827d2f58e08bc42c0cebeb5a243ca04ffed2"} Nov 02 01:14:37.740814 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:37.740775 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:37.740980 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:37.740871 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:37.740980 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:37.740959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:37.741132 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:37.741103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:37.741203 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:37.741099 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:37.741258 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:37.741207 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:37.990677 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:37.990637 2573 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="c0ccb92c8774bfe9124cacaf7b2f30a713ecc8ed272fc2f60ab6d33bb23c88a6" exitCode=0 Nov 02 01:14:37.991170 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:37.990705 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerDied","Data":"c0ccb92c8774bfe9124cacaf7b2f30a713ecc8ed272fc2f60ab6d33bb23c88a6"} Nov 02 01:14:38.996542 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:38.996509 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" event={"ID":"15670b0f-869e-49bf-9da0-9bf6ea903f2c","Type":"ContainerStarted","Data":"895f21163ba9b58b5ae0adb8d3b11410e58f803dfb454bab28dd13ad7692f946"} Nov 02 01:14:39.003281 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:39.003254 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"e9bfc61a21e27b75d224b8de1bfabf83605781f3ca15817cfe92a0bc860c1b3a"} Nov 02 01:14:39.023974 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:39.023929 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8m9jp" podStartSLOduration=8.29239 podStartE2EDuration="21.023917365s" totalImagesPullingTime="12.731527365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:39.022436814 +0000 UTC m=+21.750753742" watchObservedRunningTime="2026-11-02 01:14:39.023917365 +0000 UTC m=+21.752234293" Nov 02 01:14:39.740377 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:39.740352 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:39.740377 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:39.740359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:39.740716 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:39.740486 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:39.740716 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:39.740532 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:39.740716 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:39.740607 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:39.740716 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:39.740668 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:41.011770 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:41.011743 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" event={"ID":"4809d979-ef70-435c-b1e5-e403fa49988d","Type":"ContainerStarted","Data":"26542708b51ee31ee625fe2b6ca9139970ea97944018d66ddcdc8f4f713fae93"} Nov 02 01:14:41.739831 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:41.739641 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:41.739998 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:41.739641 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:41.739998 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:41.739924 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:41.739998 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:41.739704 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:41.739998 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:41.739987 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:41.740151 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:41.740072 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:42.014324 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.014241 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:42.014324 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.014317 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:42.031185 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.031160 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:42.049516 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.049470 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" podStartSLOduration=7.567087436 podStartE2EDuration="24.049457213s" totalImagesPullingTime="16.482369777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:42.049076982 +0000 UTC m=+24.777393900" watchObservedRunningTime="2026-11-02 01:14:42.049457213 +0000 UTC m=+24.777774131" Nov 02 01:14:42.687552 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.687482 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tqwk8"] Nov 02 01:14:42.687686 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.687620 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:42.687722 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:42.687699 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:42.690751 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.690717 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4x747"] Nov 02 01:14:42.690882 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.690849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:42.690973 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:42.690952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:42.691434 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.691410 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wx6pd"] Nov 02 01:14:42.691565 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:42.691551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:42.691688 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:42.691660 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:43.016737 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:43.016702 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:43.032327 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:43.032305 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:14:44.739579 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:44.739546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:44.739579 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:44.739569 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:44.740108 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:44.739684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:44.740108 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:44.739681 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:44.740108 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:44.739751 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:44.740108 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:44.739810 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:46.739396 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:46.739364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:46.739746 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:46.739405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:46.739746 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:46.739474 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:46.739746 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:46.739545 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:46.739746 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:46.739643 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:46.739746 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:46.739702 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:48.740308 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:48.740110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:48.740715 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:48.740213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:48.740715 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:48.740394 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4x747" podUID="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" Nov 02 01:14:48.740715 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:48.740223 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:48.740715 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:48.740456 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqwk8" podUID="dc2693cb-eac1-4cd0-b76c-db35732e9635" Nov 02 01:14:48.740715 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:48.740505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wx6pd" podUID="55f40d17-3c79-4d12-9f72-58c903869221" Nov 02 01:14:49.576026 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.576000 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeReady" Nov 02 01:14:49.576231 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.576161 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 01:14:49.629574 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.629502 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rd27f"] Nov 02 01:14:49.685452 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.685421 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lmft6"] Nov 02 01:14:49.685618 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.685537 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.689380 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.689358 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 01:14:49.689508 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.689393 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 01:14:49.689508 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.689403 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6rtc\"" Nov 02 01:14:49.695393 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.695377 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rd27f"] Nov 02 01:14:49.695471 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.695407 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmft6"] Nov 02 01:14:49.695514 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.695478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:49.700420 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.700400 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 01:14:49.700501 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.700402 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 01:14:49.700501 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.700482 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 01:14:49.700658 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.700642 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rcvqj\"" Nov 02 01:14:49.819711 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.819669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34e4cf42-f58e-442c-857c-1b7c87152cd7-config-volume\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.820096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.819719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/622df053-819a-4cbb-9ab8-bb507756ac96-cert\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:49.820096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.819739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.820096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.819838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85xq\" (UniqueName: \"kubernetes.io/projected/34e4cf42-f58e-442c-857c-1b7c87152cd7-kube-api-access-z85xq\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.820096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.819863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4cf42-f58e-442c-857c-1b7c87152cd7-tmp-dir\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.820096 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.819879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sch5m\" (UniqueName: \"kubernetes.io/projected/622df053-819a-4cbb-9ab8-bb507756ac96-kube-api-access-sch5m\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:49.921104 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4cf42-f58e-442c-857c-1b7c87152cd7-tmp-dir\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.921104 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sch5m\" (UniqueName: \"kubernetes.io/projected/622df053-819a-4cbb-9ab8-bb507756ac96-kube-api-access-sch5m\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:49.921242 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34e4cf42-f58e-442c-857c-1b7c87152cd7-config-volume\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.921242 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/622df053-819a-4cbb-9ab8-bb507756ac96-cert\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:49.921242 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.921352 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85xq\" (UniqueName: \"kubernetes.io/projected/34e4cf42-f58e-442c-857c-1b7c87152cd7-kube-api-access-z85xq\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.921385 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:49.921345 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:49.921432 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:49.921420 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:50.421400229 +0000 UTC m=+33.149717149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:14:49.921498 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921476 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4cf42-f58e-442c-857c-1b7c87152cd7-tmp-dir\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.921824 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.921803 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34e4cf42-f58e-442c-857c-1b7c87152cd7-config-volume\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.925274 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.925253 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/622df053-819a-4cbb-9ab8-bb507756ac96-cert\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:49.933676 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.933658 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85xq\" (UniqueName: \"kubernetes.io/projected/34e4cf42-f58e-442c-857c-1b7c87152cd7-kube-api-access-z85xq\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:49.934001 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:49.933986 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sch5m\" (UniqueName: \"kubernetes.io/projected/622df053-819a-4cbb-9ab8-bb507756ac96-kube-api-access-sch5m\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:50.026847 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.026806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:14:50.369307 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.369274 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmft6"] Nov 02 01:14:50.424849 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.424813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:50.424993 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:50.424942 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:50.425029 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:50.424997 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.424981383 +0000 UTC m=+34.153298305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:14:50.739860 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.739829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:50.740037 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.739886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:50.740037 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.739910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:50.744065 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.744045 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 01:14:50.744177 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.744097 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 01:14:50.745363 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.745326 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 01:14:50.745363 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.745346 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g7hgp\"" Nov 02 01:14:50.745522 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.745375 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7fmpb\"" Nov 02 01:14:50.745522 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.745403 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 01:14:50.745522 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:50.745415 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wfrhp\"" Nov 02 01:14:51.033853 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.033781 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmft6" event={"ID":"622df053-819a-4cbb-9ab8-bb507756ac96","Type":"ContainerStarted","Data":"2280d4464568344ac1429b9a44b61243a1a7c99c2141aa06576ff506a3eb182e"} Nov 02 01:14:51.404826 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.404757 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5cb8t"] Nov 02 01:14:51.416168 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.416143 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-74jqn-trlw4"] Nov 02 01:14:51.416318 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.416239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.421559 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.421532 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 01:14:51.421559 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.421554 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 01:14:51.421559 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.421544 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 01:14:51.421806 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.421592 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 01:14:51.422768 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.422749 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hdjbk\"" Nov 02 01:14:51.433223 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.433202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:14:51.433321 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.433241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:51.433321 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.433261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:51.433321 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.433280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:51.433494 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.433349 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:14:51.433494 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.433358 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:51.433494 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.433402 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:23.433388368 +0000 UTC m=+66.161705287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : secret "metrics-daemon-secret" not found Nov 02 01:14:51.433494 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.433417 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.433411006 +0000 UTC m=+36.161727912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:14:51.435653 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.435605 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:51.436009 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.435985 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:51.438696 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.438678 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5cb8t"] Nov 02 01:14:51.438777 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.438730 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-74jqn-trlw4"] Nov 02 01:14:51.438816 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.438783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.473301 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.473281 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 01:14:51.478857 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.478838 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 01:14:51.490535 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.490513 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zb852\"" Nov 02 01:14:51.533926 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.533896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.534049 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.533934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.534049 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.533971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk45f\" (UniqueName: \"kubernetes.io/projected/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-api-access-wk45f\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.534133 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.534073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.534133 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.534096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.534133 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.534111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.534224 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.534203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.534281 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.534265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a74bd44-3615-4c98-8f3b-4ba10ff959af-crio-socket\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.534317 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.534289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a74bd44-3615-4c98-8f3b-4ba10ff959af-data-volume\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.564345 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.564315 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh"] Nov 02 01:14:51.589155 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.589105 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh"] Nov 02 01:14:51.589295 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.589226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:14:51.635233 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a74bd44-3615-4c98-8f3b-4ba10ff959af-crio-socket\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635395 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a74bd44-3615-4c98-8f3b-4ba10ff959af-data-volume\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635395 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.635513 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635513 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635427 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a74bd44-3615-4c98-8f3b-4ba10ff959af-crio-socket\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635513 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk45f\" (UniqueName: \"kubernetes.io/projected/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-api-access-wk45f\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635655 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.635655 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.635655 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635581 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a74bd44-3615-4c98-8f3b-4ba10ff959af-data-volume\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635655 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.635655 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.635570 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:51.635655 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.635646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.635914 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.635702 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.135672658 +0000 UTC m=+34.863989568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:51.635914 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.635728 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:51.635914 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:51.635784 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.135768997 +0000 UTC m=+34.864085920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : configmap "kube-rbac-proxy" not found Nov 02 01:14:51.637255 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.636365 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.637255 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.636723 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.639539 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.639516 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.643906 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.643887 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ck74h\"" Nov 02 01:14:51.647909 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.647889 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk45f\" (UniqueName: \"kubernetes.io/projected/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-api-access-wk45f\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:51.648290 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.648275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.658706 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.658624 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:51.658830 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.658810 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 01:14:51.689284 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.689250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:51.706371 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.706333 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:14:51.735039 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.735014 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-t9rts"] Nov 02 01:14:51.736075 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.736054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhd8h\" (UniqueName: \"kubernetes.io/projected/5d635055-2cef-423b-af9b-9dba9996be9e-kube-api-access-fhd8h\") pod \"migrator-66df5d496b-nwfrh\" (UID: \"5d635055-2cef-423b-af9b-9dba9996be9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:14:51.752208 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.752183 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-t9rts"] Nov 02 01:14:51.755253 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.753329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.758145 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.757468 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 01:14:51.758145 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.757549 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 01:14:51.758145 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.757610 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 01:14:51.758145 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.757645 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 01:14:51.758145 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.758043 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5n6sn\"" Nov 02 01:14:51.795606 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.795579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:14:51.837401 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.837372 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58c393d2-418a-405e-b9aa-de61625fc694-signing-key\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.837527 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.837420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhd8h\" (UniqueName: \"kubernetes.io/projected/5d635055-2cef-423b-af9b-9dba9996be9e-kube-api-access-fhd8h\") pod \"migrator-66df5d496b-nwfrh\" (UID: \"5d635055-2cef-423b-af9b-9dba9996be9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:14:51.837527 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.837495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qwq\" (UniqueName: \"kubernetes.io/projected/58c393d2-418a-405e-b9aa-de61625fc694-kube-api-access-82qwq\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.837601 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.837532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58c393d2-418a-405e-b9aa-de61625fc694-signing-cabundle\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.857630 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.857605 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhd8h\" (UniqueName: \"kubernetes.io/projected/5d635055-2cef-423b-af9b-9dba9996be9e-kube-api-access-fhd8h\") pod \"migrator-66df5d496b-nwfrh\" (UID: \"5d635055-2cef-423b-af9b-9dba9996be9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:14:51.910709 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.910642 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:14:51.938734 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.938694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58c393d2-418a-405e-b9aa-de61625fc694-signing-key\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.938734 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.938737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qwq\" (UniqueName: \"kubernetes.io/projected/58c393d2-418a-405e-b9aa-de61625fc694-kube-api-access-82qwq\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.938907 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.938756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58c393d2-418a-405e-b9aa-de61625fc694-signing-cabundle\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.939333 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.939312 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58c393d2-418a-405e-b9aa-de61625fc694-signing-cabundle\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.941316 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.941298 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58c393d2-418a-405e-b9aa-de61625fc694-signing-key\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.951223 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.951203 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qwq\" (UniqueName: \"kubernetes.io/projected/58c393d2-418a-405e-b9aa-de61625fc694-kube-api-access-82qwq\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:51.986309 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:51.986280 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4x747"] Nov 02 01:14:52.016599 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.016528 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tqwk8"] Nov 02 01:14:52.020293 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:52.020262 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2693cb_eac1_4cd0_b76c_db35732e9635.slice/crio-c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87 WatchSource:0}: Error finding container c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87: Status 404 returned error can't find the container with id c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87 Nov 02 01:14:52.037009 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.036980 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tqwk8" event={"ID":"dc2693cb-eac1-4cd0-b76c-db35732e9635","Type":"ContainerStarted","Data":"c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87"} Nov 02 01:14:52.038224 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.038198 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4x747" event={"ID":"f6ca2c0e-a09c-4c72-bb67-c595dad9050c","Type":"ContainerStarted","Data":"2edb5feaff3c6771ae4c27a7899470e629bd0255467e44845188cae906260c91"} Nov 02 01:14:52.079018 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.078989 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-74jqn-trlw4"] Nov 02 01:14:52.081692 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:52.081665 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ca99b_cdd2_4659_bd4d_839c69e15f46.slice/crio-145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2 WatchSource:0}: Error finding container 145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2: Status 404 returned error can't find the container with id 145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2 Nov 02 01:14:52.100261 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.100237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:14:52.140262 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.140234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:52.140352 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.140307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:52.140403 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:52.140390 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:52.140459 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:52.140450 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.140436107 +0000 UTC m=+35.868753027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : configmap "kube-rbac-proxy" not found Nov 02 01:14:52.140521 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:52.140480 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:52.140580 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:52.140545 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.140526026 +0000 UTC m=+35.868842948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:52.205599 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.205568 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh"] Nov 02 01:14:52.212298 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:52.212270 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d635055_2cef_423b_af9b_9dba9996be9e.slice/crio-59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc WatchSource:0}: Error finding container 59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc: Status 404 returned error can't find the container with id 59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc Nov 02 01:14:52.405874 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:52.405839 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-t9rts"] Nov 02 01:14:52.409355 ip-10-0-137-1 kubenswrapper[2573]: W1102 01:14:52.409324 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c393d2_418a_405e_b9aa_de61625fc694.slice/crio-bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756 WatchSource:0}: Error finding container bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756: Status 404 returned error can't find the container with id bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756 Nov 02 01:14:53.046913 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:53.046876 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" event={"ID":"58c393d2-418a-405e-b9aa-de61625fc694","Type":"ContainerStarted","Data":"bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756"} Nov 02 01:14:53.051941 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:53.051885 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" event={"ID":"5d635055-2cef-423b-af9b-9dba9996be9e","Type":"ContainerStarted","Data":"59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc"} Nov 02 01:14:53.054020 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:53.053967 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-74jqn-trlw4" event={"ID":"844ca99b-cdd2-4659-bd4d-839c69e15f46","Type":"ContainerStarted","Data":"145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2"} Nov 02 01:14:53.149802 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:53.148995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:53.149802 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:53.149101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:53.149802 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:53.149221 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:53.149802 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:53.149283 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:55.149265045 +0000 UTC m=+37.877581954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : configmap "kube-rbac-proxy" not found Nov 02 01:14:53.149802 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:53.149694 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:53.149802 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:53.149758 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:55.149742414 +0000 UTC m=+37.878059322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:53.452222 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:53.452063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:53.452377 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:53.452258 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:53.452377 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:53.452346 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.452324977 +0000 UTC m=+40.180641899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:14:55.065335 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:55.065295 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" event={"ID":"5d635055-2cef-423b-af9b-9dba9996be9e","Type":"ContainerStarted","Data":"246ab6cbb7d7965787bb0bf27a91fcbcc5865389aeba0cdf7baa6440abc3abcf"} Nov 02 01:14:55.169392 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:55.169359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:55.169566 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:55.169457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:55.169566 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:55.169548 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:55.169675 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:55.169601 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.16958585 +0000 UTC m=+41.897902769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : configmap "kube-rbac-proxy" not found Nov 02 01:14:55.169907 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:55.169880 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:55.170038 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:55.169951 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.169935286 +0000 UTC m=+41.898252210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:55.319368 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:55.319277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qjmzq_31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa/konnectivity-agent/0.log" limit=1024 Nov 02 01:14:56.071697 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:56.071661 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmft6" event={"ID":"622df053-819a-4cbb-9ab8-bb507756ac96","Type":"ContainerStarted","Data":"bbb01f0850f46f32dad611cc81b4714494e2e839dc58648a41f3f3ec9be6a0b8"} Nov 02 01:14:56.091082 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:56.090529 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" podStartSLOduration=2.611608566 podStartE2EDuration="5.090513503s" totalImagesPullingTime="2.478904937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:55.097449752 +0000 UTC m=+37.825766680" watchObservedRunningTime="2026-11-02 01:14:56.090513503 +0000 UTC m=+38.818830433" Nov 02 01:14:56.091082 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:56.090886 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lmft6" podStartSLOduration=1.5815202400000001 podStartE2EDuration="7.090873085s" totalImagesPullingTime="5.509352845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:56.090429064 +0000 UTC m=+38.818745993" watchObservedRunningTime="2026-11-02 01:14:56.090873085 +0000 UTC m=+38.819190013" Nov 02 01:14:57.077024 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:57.076998 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" event={"ID":"58c393d2-418a-405e-b9aa-de61625fc694","Type":"ContainerStarted","Data":"4b3946afa754e10d8d565df5d77c34a02793bc610facb2f93bf5efd139c24b7c"} Nov 02 01:14:57.496708 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:57.496664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:14:57.496892 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:57.496836 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:57.496952 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:57.496917 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:05.496895529 +0000 UTC m=+48.225212449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:14:58.082490 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:58.082450 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4x747" event={"ID":"f6ca2c0e-a09c-4c72-bb67-c595dad9050c","Type":"ContainerStarted","Data":"9ba6c33e1b5d03e58f01ee200b0fe9c504718f0a005ea6644f02c558fbfe1211"} Nov 02 01:14:58.084351 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:58.082584 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:14:58.084351 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:58.084311 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-74jqn-trlw4" event={"ID":"844ca99b-cdd2-4659-bd4d-839c69e15f46","Type":"ContainerStarted","Data":"4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c"} Nov 02 01:14:58.104805 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:58.104743 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4x747" podStartSLOduration=34.752380468 podStartE2EDuration="40.104722985s" totalImagesPullingTime="5.352342517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:58.103510782 +0000 UTC m=+40.831827705" watchObservedRunningTime="2026-11-02 01:14:58.104722985 +0000 UTC m=+40.833039914" Nov 02 01:14:58.124605 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:58.124555 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-74jqn-trlw4" podStartSLOduration=2.199912546 podStartE2EDuration="7.124537778s" totalImagesPullingTime="4.924625232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:58.122744271 +0000 UTC m=+40.851061190" watchObservedRunningTime="2026-11-02 01:14:58.124537778 +0000 UTC m=+40.852854766" Nov 02 01:14:59.213241 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:59.213203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:59.213709 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:59.213295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:59.213709 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:59.213426 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:59.213709 ip-10-0-137-1 kubenswrapper[2573]: E1102 01:14:59.213509 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:15:07.213489108 +0000 UTC m=+49.941806076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:59.213833 ip-10-0-137-1 kubenswrapper[2573]: I1102 01:14:59.213760 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:14:59.830155 ip-10-0-137-1 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 01:14:59.837841 ip-10-0-137-1 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 01:14:59.838008 ip-10-0-137-1 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 01:14:59.838056 ip-10-0-137-1 systemd[1]: kubelet.service: Consumed 2.532s CPU time. Nov 02 01:14:59.853795 ip-10-0-137-1 systemd[1]: Starting Kubernetes Kubelet... Nov 02 01:14:59.946457 ip-10-0-137-1 kubenswrapper[4552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:59.946457 ip-10-0-137-1 kubenswrapper[4552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 01:14:59.946457 ip-10-0-137-1 kubenswrapper[4552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:59.946457 ip-10-0-137-1 kubenswrapper[4552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:59.948547 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948460 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.948547 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948548 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948555 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948558 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948561 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948565 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948567 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948571 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948573 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948576 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948579 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948582 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948584 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948587 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948590 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948593 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948595 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948598 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948601 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948603 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948606 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.948622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948610 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948613 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948621 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948625 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948628 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948631 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948634 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948636 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948639 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948642 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948645 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948647 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948650 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948653 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948655 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948658 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948660 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948663 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948665 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.949112 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948668 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948670 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948673 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948676 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948678 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948681 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948683 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948686 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948689 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948691 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948694 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948696 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948699 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948702 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948707 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948712 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948715 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948718 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948720 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.949591 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948723 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948726 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948729 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948731 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948734 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948737 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948740 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948744 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948748 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948751 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948754 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948757 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948759 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948762 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948765 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948768 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948771 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948773 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948777 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.950102 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948779 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948782 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948785 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948787 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948790 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948793 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948796 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948798 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948801 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948803 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948806 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948809 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948811 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948814 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948816 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948819 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948822 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948824 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948827 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.950592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948829 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948832 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948834 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948837 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948840 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948843 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948846 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948850 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.948854 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949646 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949654 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949657 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949661 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949665 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949668 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949671 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949674 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949677 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949679 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949684 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.951059 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949688 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949691 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949694 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949696 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949699 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949702 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949705 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949707 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949711 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949715 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949718 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949721 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949724 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949727 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949730 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949733 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949736 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949738 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949741 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.951579 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949745 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949748 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949751 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949753 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949756 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949759 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949772 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949775 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949778 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949781 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949783 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949786 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949789 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949798 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949801 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949804 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949806 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949809 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949812 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949815 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.952045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949818 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949821 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949824 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949826 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949829 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949832 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949834 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949837 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949840 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949842 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949845 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949848 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949850 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949853 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949856 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949859 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949863 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949865 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949868 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.952552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949870 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949873 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949876 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949878 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949881 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949883 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949886 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949889 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949891 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949894 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949897 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949900 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949902 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949905 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949908 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949910 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949913 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949915 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949918 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.953055 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949920 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949923 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949925 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949928 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949931 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949934 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949936 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949939 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949941 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949944 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949950 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949952 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949955 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949957 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949960 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949963 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949965 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.949970 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950486 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950491 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.953545 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950494 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950497 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950500 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950503 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950506 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950508 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950511 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950514 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950517 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950520 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950522 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950525 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950528 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950531 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950534 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950537 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950540 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950542 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950545 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950547 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.954041 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950550 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950552 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950556 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950558 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950561 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950564 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950566 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950569 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950572 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950575 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950577 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950580 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950583 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950585 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950588 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950590 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950593 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950596 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950599 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950601 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.954593 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950604 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950607 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950610 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950612 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950615 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950618 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950621 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950623 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950626 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950629 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950631 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950634 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950637 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950639 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950644 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950648 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950651 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950654 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950657 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.955067 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950660 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950665 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950667 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950670 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950672 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950675 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950678 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950680 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950683 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950685 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950688 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950691 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950694 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950696 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950699 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950701 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950704 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950706 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950710 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.955592 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950712 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950715 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950717 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950720 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950723 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950726 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950729 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950731 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950736 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950739 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950743 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950746 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950749 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950751 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950755 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950758 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950761 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950763 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950766 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.956177 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950769 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950772 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950775 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950777 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950780 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950782 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.950785 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950888 4552 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950898 4552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950907 4552 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950913 4552 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950917 4552 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950921 4552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950928 4552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950932 4552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950936 4552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950939 4552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950943 4552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950946 4552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950950 4552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950952 4552 flags.go:64] FLAG: --cgroup-root="" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950955 4552 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950958 4552 flags.go:64] FLAG: --client-ca-file="" Nov 02 01:14:59.956639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950961 4552 flags.go:64] FLAG: --cloud-provider="external" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950965 4552 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950991 4552 flags.go:64] FLAG: --cluster-domain="" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950995 4552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.950999 4552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951003 4552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951006 4552 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951010 4552 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951013 4552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951016 4552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951019 4552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951023 4552 flags.go:64] FLAG: --contention-profiling="false" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951026 4552 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951029 4552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951032 4552 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951035 4552 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951040 4552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951043 4552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951046 4552 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951049 4552 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951052 4552 flags.go:64] FLAG: --enable-server="true" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951055 4552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951067 4552 flags.go:64] FLAG: --event-burst="100" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951070 4552 flags.go:64] FLAG: --event-qps="50" Nov 02 01:14:59.957252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951076 4552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951080 4552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951083 4552 flags.go:64] FLAG: --eviction-hard="" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951087 4552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951090 4552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951093 4552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951096 4552 flags.go:64] FLAG: --eviction-soft="" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951100 4552 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951103 4552 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951106 4552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951110 4552 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951127 4552 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951131 4552 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951134 4552 flags.go:64] FLAG: --feature-gates="" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951139 4552 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951142 4552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951145 4552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951148 4552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951151 4552 flags.go:64] FLAG: --healthz-port="10248" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951154 4552 flags.go:64] FLAG: --help="false" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951157 4552 flags.go:64] FLAG: --hostname-override="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951160 4552 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951163 4552 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 01:14:59.957890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951166 4552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951170 4552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951173 4552 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951176 4552 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951179 4552 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951181 4552 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951184 4552 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951187 4552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951190 4552 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951195 4552 flags.go:64] FLAG: --kube-reserved="" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951198 4552 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951201 4552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951204 4552 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951206 4552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951209 4552 flags.go:64] FLAG: --lock-file="" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951212 4552 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951215 4552 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951218 4552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951224 4552 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951227 4552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951230 4552 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951237 4552 flags.go:64] FLAG: --logging-format="text" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951240 4552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951243 4552 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 01:14:59.958478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951246 4552 flags.go:64] FLAG: --manifest-url="" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951249 4552 flags.go:64] FLAG: --manifest-url-header="" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951254 4552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951257 4552 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951261 4552 flags.go:64] FLAG: --max-pods="110" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951265 4552 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951268 4552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951271 4552 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951274 4552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951277 4552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951280 4552 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951283 4552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951290 4552 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951293 4552 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951296 4552 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951299 4552 flags.go:64] FLAG: --pod-cidr="" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951302 4552 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951307 4552 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951310 4552 flags.go:64] FLAG: --pods-per-core="0" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951313 4552 flags.go:64] FLAG: --port="10250" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951316 4552 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951319 4552 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-025a43f96557ca686" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951322 4552 flags.go:64] FLAG: --qos-reserved="" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951325 4552 flags.go:64] FLAG: --read-only-port="10255" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951328 4552 flags.go:64] FLAG: --register-node="true" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951331 4552 flags.go:64] FLAG: --register-with-taints="" Nov 02 01:14:59.959066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951335 4552 flags.go:64] FLAG: --registry-burst="10" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951337 4552 flags.go:64] FLAG: --registry-qps="5" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951340 4552 flags.go:64] FLAG: --reserved-cpus="" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951345 4552 flags.go:64] FLAG: --reserved-memory="" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951349 4552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951352 4552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951356 4552 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951359 4552 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951362 4552 flags.go:64] FLAG: --runonce="false" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951365 4552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951368 4552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951371 4552 flags.go:64] FLAG: --seccomp-default="false" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951374 4552 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951377 4552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951380 4552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951383 4552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951386 4552 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951389 4552 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951392 4552 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951395 4552 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951398 4552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951401 4552 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951404 4552 flags.go:64] FLAG: --system-cgroups="" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951408 4552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951413 4552 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 01:14:59.959725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951416 4552 flags.go:64] FLAG: --tls-cert-file="" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951419 4552 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951423 4552 flags.go:64] FLAG: --tls-min-version="" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951426 4552 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951429 4552 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951432 4552 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951434 4552 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951437 4552 flags.go:64] FLAG: --v="2" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951442 4552 flags.go:64] FLAG: --version="false" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951446 4552 flags.go:64] FLAG: --vmodule="" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951451 4552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951470 4552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951605 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951609 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951612 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951615 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951618 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951621 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951623 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951626 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951629 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951632 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951634 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.960349 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951637 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951639 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951642 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951644 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951647 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951649 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951652 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951656 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951658 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951661 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951664 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951667 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951670 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951672 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951675 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951677 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951680 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951684 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951688 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.960914 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951692 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951695 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951698 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951701 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951704 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951706 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951709 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951712 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951714 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951717 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951719 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951722 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951725 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951727 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951730 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951733 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951735 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951738 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951740 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.961378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951743 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951747 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951750 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951752 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951755 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951758 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951760 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951763 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951766 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951768 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951771 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951774 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951777 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951780 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951783 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951786 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951789 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951792 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951794 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951797 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.961843 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951800 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951803 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951805 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951808 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951811 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951813 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951816 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951818 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951821 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951823 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951826 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951829 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951832 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951836 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951838 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951841 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951844 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951846 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951849 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951852 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.962387 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951854 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951857 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951859 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951862 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951865 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951869 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951872 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951876 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951879 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951883 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951886 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951889 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951892 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951895 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951897 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951900 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.962880 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951902 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.951928 4552 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951969 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951974 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951977 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951980 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951984 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951987 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951990 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951993 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.951998 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952001 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952004 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952007 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952009 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952012 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.963285 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952015 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952018 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952021 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952023 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952026 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952030 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952033 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952035 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952038 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952041 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952044 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952047 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952049 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952052 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952055 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952057 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952060 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952063 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952065 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952068 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.963916 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952070 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952073 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952076 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952079 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952081 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952084 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952088 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952090 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952094 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952099 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952101 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952104 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952109 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952127 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952130 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952133 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952135 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952139 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.964719 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952142 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952145 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952147 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952150 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952153 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952156 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952159 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952161 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952164 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952167 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952169 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952172 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952174 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952177 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952179 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952182 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952185 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952187 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952190 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.965343 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952192 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952196 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952199 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952202 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952204 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952207 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952209 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952212 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952215 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952217 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952220 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952223 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952227 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952230 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952232 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952235 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952238 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952240 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952244 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952247 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.965901 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952250 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952252 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952255 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952258 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952261 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952264 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952267 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952269 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952272 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952274 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952277 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952280 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952282 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952286 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.952289 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.957224 4552 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.957243 4552 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.957266 4552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.957273 4552 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 01:14:59.966548 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957385 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957393 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957399 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957403 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957409 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957413 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957418 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957422 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957426 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957431 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957434 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957439 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957451 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957456 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957460 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957464 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957468 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957472 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957476 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.967263 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957480 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957484 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957488 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957493 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957497 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957500 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957504 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957508 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957513 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957520 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957526 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957530 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957535 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957539 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957543 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957547 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957551 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957555 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957559 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.967865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957563 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957568 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957571 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957575 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957579 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957584 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957588 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957599 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957603 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957607 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957611 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957615 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957619 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957624 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957628 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957632 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957635 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957639 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957643 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957647 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.968374 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957651 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957655 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957660 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957664 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957668 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957671 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957675 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957679 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957683 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957687 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957691 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957695 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957699 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957703 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957707 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957711 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957715 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957719 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957723 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.968929 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957727 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957738 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957742 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957747 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957751 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957755 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957769 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957776 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957780 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957784 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957789 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957793 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957797 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957801 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957805 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957809 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957813 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957817 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957828 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957832 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.969813 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957836 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957840 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957844 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957848 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957853 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957857 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957861 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957865 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.957870 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.957910 4552 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958220 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958227 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958233 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958238 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958249 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958253 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.970384 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958257 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958261 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958266 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958270 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958274 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958278 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958282 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958286 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958290 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958294 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958298 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958302 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958306 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958310 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958314 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958319 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958323 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958328 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958332 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958336 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.970782 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958340 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958344 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958348 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958352 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958357 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958361 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958365 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958369 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958373 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958377 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958382 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958393 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958398 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958402 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958406 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958412 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958418 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958422 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958427 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.971321 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958432 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958436 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958440 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958444 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958449 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958453 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958458 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958462 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958466 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958471 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958475 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958479 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958483 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958488 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958492 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958496 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958501 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958505 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958509 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958513 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.971772 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958517 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958521 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958525 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958530 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958534 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958544 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958549 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958553 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958557 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958561 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958565 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958571 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958577 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958581 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958597 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958601 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958605 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958609 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958613 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.972352 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958617 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958621 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958626 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958630 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958634 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958638 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958642 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958646 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958651 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958655 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958659 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958663 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958667 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958671 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958676 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958680 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958684 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958688 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958693 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958708 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.972854 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958712 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958717 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.958751 4552 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958818 4552 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958825 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958829 4552 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958834 4552 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958838 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958843 4552 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958847 4552 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958851 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958855 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958859 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958864 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958870 4552 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:59.973463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958876 4552 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958882 4552 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958887 4552 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958892 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958896 4552 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958901 4552 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958905 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958909 4552 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958914 4552 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958918 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958922 4552 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958926 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958931 4552 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958935 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958939 4552 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958943 4552 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958947 4552 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958962 4552 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958967 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958971 4552 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:59.973820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958975 4552 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958979 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958983 4552 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958987 4552 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958991 4552 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.958995 4552 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959000 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959004 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959008 4552 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959012 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959016 4552 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959020 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959025 4552 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959029 4552 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959035 4552 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959041 4552 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959046 4552 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959050 4552 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959054 4552 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:59.974314 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959058 4552 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959062 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959066 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959071 4552 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959075 4552 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959079 4552 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959084 4552 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959088 4552 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959092 4552 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959096 4552 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959101 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959127 4552 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959132 4552 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959136 4552 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959140 4552 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959144 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959149 4552 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959154 4552 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959158 4552 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959162 4552 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:59.974757 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959166 4552 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959170 4552 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959174 4552 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959178 4552 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959182 4552 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959186 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959191 4552 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959195 4552 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959199 4552 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959203 4552 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959207 4552 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959211 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959215 4552 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959219 4552 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959223 4552 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959227 4552 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959231 4552 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959235 4552 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959239 4552 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:59.975243 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959243 4552 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959247 4552 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959252 4552 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959256 4552 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959261 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959272 4552 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959276 4552 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959280 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959286 4552 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959293 4552 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959297 4552 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959301 4552 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959305 4552 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959309 4552 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959313 4552 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:14:59.959318 4552 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.959391 4552 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.960901 4552 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.960963 4552 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 01:14:59.975714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.961250 4552 server.go:1014] "Starting client certificate rotation" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.961367 4552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.961416 4552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:09:18 +0000 UTC" deadline="2028-05-12 01:38:25.456226362 +0000 UTC" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.961487 4552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13368h23m25.494741923s" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.962009 4552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.964869 4552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.966905 4552 log.go:25] "Validated CRI v1 runtime API" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.968953 4552 log.go:25] "Validated CRI v1 image API" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.969772 4552 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 01:14:59.976190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.971943 4552 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a1c5980-4a96-4861-ac74-e39d4d89d2ac:/dev/nvme0n1p3 a4741b66-d346-4584-a51f-21ad1107403a:/dev/nvme0n1p4] Nov 02 01:14:59.976431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.971963 4552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2280d4464568344ac1429b9a44b61243a1a7c99c2141aa06576ff506a3eb182e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2280d4464568344ac1429b9a44b61243a1a7c99c2141aa06576ff506a3eb182e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2edb5feaff3c6771ae4c27a7899470e629bd0255467e44845188cae906260c91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2edb5feaff3c6771ae4c27a7899470e629bd0255467e44845188cae906260c91/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0e4f79ddc8a675e4ceede6fd99a69b7f24362c40ca650fd1cfd34adb614521e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0e4f79ddc8a675e4ceede6fd99a69b7f24362c40ca650fd1cfd34adb614521e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8/volumes/kubernetes.io~projected/kube-api-access-8xx9j:{mountpoint:/var/lib/kubelet/pods/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8/volumes/kubernetes.io~projected/kube-api-access-8xx9j major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a74bd44-3615-4c98-8f3b-4ba10ff959af/volumes/kubernetes.io~projected/kube-api-access-wk45f:{mountpoint:/var/lib/kubelet/pods/0a74bd44-3615-4c98-8f3b-4ba10ff959af/volumes/kubernetes.io~projected/kube-api-access-wk45f major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15670b0f-869e-49bf-9da0-9bf6ea903f2c/volumes/kubernetes.io~projected/kube-api-access-mv252:{mountpoint:/var/lib/kubelet/pods/15670b0f-869e-49bf-9da0-9bf6ea903f2c/volumes/kubernetes.io~projected/kube-api-access-mv252 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d617c8-7375-4b58-93b2-7eb39c4bb8c0/volumes/kubernetes.io~projected/kube-api-access-f7rdt:{mountpoint:/var/lib/kubelet/pods/30d617c8-7375-4b58-93b2-7eb39c4bb8c0/volumes/kubernetes.io~projected/kube-api-access-f7rdt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e4cf42-f58e-442c-857c-1b7c87152cd7/volumes/kubernetes.io~projected/kube-api-access-z85xq:{mountpoint:/var/lib/kubelet/pods/34e4cf42-f58e-442c-857c-1b7c87152cd7/volumes/kubernetes.io~projected/kube-api-access-z85xq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volumes/kubernetes.io~projected/kube-api-access-l72cd:{mountpoint:/var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volumes/kubernetes.io~projected/kube-api-access-l72cd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f40d17-3c79-4d12-9f72-58c903869221/volumes/kubernetes.io~projected/kube-api-access-cz8qd:{mountpoint:/var/lib/kubelet/pods/55f40d17-3c79-4d12-9f72-58c903869221/volumes/kubernetes.io~projected/kube-api-access-cz8qd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58c393d2-418a-405e-b9aa-de61625fc694/volumes/kubernetes.io~projected/kube-api-access-82qwq:{mountpoint:/var/lib/kubelet/pods/58c393d2-418a-405e-b9aa-de61625fc694/volumes/kubernetes.io~projected/kube-api-access-82qwq major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58c393d2-418a-405e-b9aa-de61625fc694/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/58c393d2-418a-405e-b9aa-de61625fc694/volumes/kubernetes.io~secret/signing-key major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d635055-2cef-423b-af9b-9dba9996be9e/volumes/kubernetes.io~projected/kube-api-access-fhd8h:{mountpoint:/var/lib/kubelet/pods/5d635055-2cef-423b-af9b-9dba9996be9e/volumes/kubernetes.io~projected/kube-api-access-fhd8h major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/622df053-819a-4cbb-9ab8-bb507756ac96/volumes/kubernetes.io~projected/kube-api-access-sch5m:{mountpoint:/var/lib/kubelet/pods/622df053-819a-4cbb-9ab8-bb507756ac96/volumes/kubernetes.io~projected/kube-api-access-sch5m major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/622df053-819a-4cbb-9ab8-bb507756ac96/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/622df053-819a-4cbb-9ab8-bb507756ac96/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cfdec63-7004-4a85-91dd-6a7a6db01761/volumes/kubernetes.io~projected/kube-api-access-zsd8x:{mountpoint:/var/lib/kubelet/pods/6cfdec63-7004-4a85-91dd-6a7a6db01761/volumes/kubernetes.io~projected/kube-api-access-zsd8x major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/779489ef-236e-4998-a739-964989ab5d07/volumes/kubernetes.io~projected/kube-api-access-gk4lp:{mountpoint:/var/lib/kubelet/pods/779489ef-236e-4998-a739-964989ab5d07/volumes/kubernetes.io~projected/kube-api-access-gk4lp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/844ca99b-cdd2-4659-bd4d-839c69e15f46/volumes/kubernetes.io~projected/kube-api-access-4ztjw:{mountpoint:/var/lib/kubelet/pods/844ca99b-cdd2-4659-bd4d-839c69e15f46/volumes/kubernetes.io~projected/kube-api-access-4ztjw major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5aaf1d9-2a69-4a0f-9503-2217191d7131/volumes/kubernetes.io~projected/kube-api-access-46rsb:{mountpoint:/var/lib/kubelet/pods/d5aaf1d9-2a69-4a0f-9503-2217191d7131/volumes/kubernetes.io~projected/kube-api-access-46rsb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc2693cb-eac1-4cd0-b76c-db35732e9635/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dc2693cb-eac1-4cd0-b76c-db35732e9635/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~projected/kube-api-access-dt9j5:{mountpoint:/var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~projected/kube-api-access-dt9j5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6ca2c0e-a09c-4c72-bb67-c595dad9050c/volumes/kubernetes.io~projected/kube-api-access-fr7fg:{mountpoint:/var/lib/kubelet/pods/f6ca2c0e-a09c-4c72-bb67-c595dad9050c/volumes/kubernetes.io~projected/kube-api-access-fr7fg major:0 minor:206 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4416e45ed43ffd1e8fe649d828dec6460c655c0ce783a18325e6de78fd51e2d5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e27a1fb5aa1bd4005e2022bea57749e9f6adfdbb88bf459da81539d1c728bf57/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1c3ffd9dfd8a84d606a5d214a2d185b019b643e034a4da05bb4d0325aef7ea56/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/be28c5804f32d8b93c5fc6c5c017f2adb7104364b2fa744d63ca612629d4a5ba/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/7f30d0a6e20e3258ea5674759f3168448914ec1e1c36ac73b05f423edf7f7d0e/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2e70834ff23f8f931d3c4f54a62a3146c039a1633955a4a46a8cd7f29b3f2a16/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/01fa1e281757a5b34fc348be8d3269035800da2bbb9b077ac6f58fb669c863d9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/27643f662449e43a6519157cc5ad9aa591cbb187f0c1463ba9d7f886f2a6938c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5dc1485d43087c51be89c59e7223e45b3ce17ad21e7af44482885a120a39fc91/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/841ad12bf099150265aa6e4dcee65b31531a0813f77129bc228463a2247a31a7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/33a6955f68e905b498ceefec8164b351a441e75a506d9563e7a972f91e40d74d/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/7dc8450a0d5e289d116cb8c9ff4bafe0b40cff9d2cb9dfb6c74ee469b272860a/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3c0b0fb1ca9fd899880d759239b9e680610a967eab17810a7b39663f968293b0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/26e5c01deba63b137e9693460440d798ad373c9bea041c9fb1d6bb4a96297630/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/1e6e0ad6f550530c59ddd652bd2b8bfb3784c25f04710e43abdcfcf6954039dd/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/53a4e6a44a24c9bba2f24fefe2a1e6e11aaa3f8dd6b52a0582125e6dbfc7b5c8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f068052e48de69a504b6e562cf801f401ab51d649d85ef5778928234988506f2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/52266fd21d0baadeb482a8f7ce67ca94ed44b3d954b67fd870843ddb39efaeac/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b9109793eb9a6b57e6bd5581f2702b6f964c087b0ed82a49324e28bdb04afce2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/93aba21acd5cb4f456d852b03dfa995a47a9890639705619ddf72974c40ab79f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8ec060b376d786cbf42f97a7d7e8c84d2ddf6e9607c6132d490c4987cb7c9751/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/f7e17ad99e34183e1e054b92b88d7862be4c59c2743256649ea1767ea71da160/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/40c3fe7e9acea8fdbd85afc48cd0bce15799923b12889e4e656b9b6d4d8cf782/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/13a0d0771cf853275bb26768164cca784aa2763a2bdf2db78dbbb496832ad503/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/bd8ddd212318797abeb3ad104cef9025fe12eb6f1c477e0e4f02faef4be87ae0/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/b1ca9976bd86c87d065d7fdc35d432c9e4ca3d72cf978741c7253ceecf66ee75/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/eeb4503fd1edbe29fbc3c174158408b93557a9135161ff160686b7d7cf1050e8/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/c90210b50d6b1c5c32dd0ac65ace2138c9b05a5840dc4971cf070ac7dcdf2436/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/82ec1dcb81466c00d805886ee9ddd8aaaacf285dc46943cf1c6a0110e82c8b65/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d4f060d3beedf50ebb249cc0359ce83fbc7b03ed600ce90457e6f381db5454e9/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2dcf08556839f04f352820a046b395d2135d0fdc7797fbd29b1d0b354a05c3c2/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/825ddf182ef8c1d93142ac90f7fa13ee8a9a6aec6328a3ae8b307f0611034cce/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/a84ef84586f88638f46d0b7caba31473df01a7885eff3147a3e1d45aec3011b4/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8c6d57eed5e96a4f6d6bf894b6dafc8d05e79d180bdcec60c5f4bf64fd795ece/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/74b671ff5305da0c1f58b5319e3f39a051b7abc4c22796d42222ab3f7cd4c2fb/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/788ab94d47a0df544151500406778d7cdf85a6f8b0b182fd162b46c64fed3603/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9f6af3820ecc134278d8d8690bfee64ade38b5ff6a217a7098c779af0b0d899f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0a0ea5fbe314f6a378a44b049200ca976493dd953cd3db548eb6e4555ee62153/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c18139e17ddf6335bdd3238dacfdaed9740020864f197c5e1fc01a00bc12732c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c6fc301b0b1ba58a64eb66b06af28944495a83a12490bea385db48613a897817/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f60127838c125fd838749fe308e3edd75d6a985ebd0bd3338486d7edea97af61/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/284bdca1c3dcd4b7e41749ee09cb05177e1f3a313b9b692a50fbde41ae2c840d/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5b86b32d419507626cdc61aee84d0cfc8d1c13d368687bfb0593a2093b3ddfea/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 01:14:59.981249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.980792 4552 manager.go:217] Machine: {Timestamp:2026-11-02 01:14:59.979961146 +0000 UTC m=+0.085750053 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100076 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c4a499b421177b16f38fb93bb757 SystemUUID:ec22c4a4-99b4-2117-7b16-f38fb93bb757 BootID:60a04206-17f5-43b3-9423-b229ebad30eb Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8/volumes/kubernetes.io~projected/kube-api-access-8xx9j DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a74bd44-3615-4c98-8f3b-4ba10ff959af/volumes/kubernetes.io~projected/kube-api-access-wk45f DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/55f40d17-3c79-4d12-9f72-58c903869221/volumes/kubernetes.io~projected/kube-api-access-cz8qd DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6cfdec63-7004-4a85-91dd-6a7a6db01761/volumes/kubernetes.io~projected/kube-api-access-zsd8x DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c97d19eaf266d95fc7958e9dc558909d6c23177c015a25a654df1e06e3b4f18/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/622df053-819a-4cbb-9ab8-bb507756ac96/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/779489ef-236e-4998-a739-964989ab5d07/volumes/kubernetes.io~projected/kube-api-access-gk4lp DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2f1d36e01bc0fa50ba2204b564069b99d2f055bf0e8e4def0a4ffcfd35df595/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/844ca99b-cdd2-4659-bd4d-839c69e15f46/volumes/kubernetes.io~projected/kube-api-access-4ztjw DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/15670b0f-869e-49bf-9da0-9bf6ea903f2c/volumes/kubernetes.io~projected/kube-api-access-mv252 DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2edb5feaff3c6771ae4c27a7899470e629bd0255467e44845188cae906260c91/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58c393d2-418a-405e-b9aa-de61625fc694/volumes/kubernetes.io~projected/kube-api-access-82qwq DeviceMajor:0 DeviceMinor:220 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd7fe9697c5383a2c1f607ca392fde1e48c80c2ceec308d000aafd06949bc756/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58c393d2-418a-405e-b9aa-de61625fc694/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:219 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volumes/kubernetes.io~projected/kube-api-access-l72cd DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2280d4464568344ac1429b9a44b61243a1a7c99c2141aa06576ff506a3eb182e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4809d979-ef70-435c-b1e5-e403fa49988d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb4f89b9695231129923372c2cf51dc7fc5bd77736ebf668c63d3bb7fce73f1f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0e658af8bb9fc57faddf6bb2711b9cf75b3636e9129aed8a354e23eab8735fe/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c76f53a27f3c51f947947c9270695666574d6bb1f057cbbb51839bd278e17f87/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6ca2c0e-a09c-4c72-bb67-c595dad9050c/volumes/kubernetes.io~projected/kube-api-access-fr7fg DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5d635055-2cef-423b-af9b-9dba9996be9e/volumes/kubernetes.io~projected/kube-api-access-fhd8h DeviceMajor:0 DeviceMinor:216 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~projected/kube-api-access-dt9j5 DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/30d617c8-7375-4b58-93b2-7eb39c4bb8c0/volumes/kubernetes.io~projected/kube-api-access-f7rdt DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb01c52331baecdffa64ccdb6f0b38d79c8f4b597560abf7546c82e370481759/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc2693cb-eac1-4cd0-b76c-db35732e9635/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d5aaf1d9-2a69-4a0f-9503-2217191d7131/volumes/kubernetes.io~projected/kube-api-access-46rsb DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cd2f888654ab4f0782dbdeeba5a55d5012dfa6b5fd8a44bd6e72aa5765dc95c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34e4cf42-f58e-442c-857c-1b7c87152cd7/volumes/kubernetes.io~projected/kube-api-access-z85xq DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/622df053-819a-4cbb-9ab8-bb507756ac96/volumes/kubernetes.io~projected/kube-api-access-sch5m DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59b05ffd230698bcc460df4c47e2bcf58991113995ebc352c1e0f79d296edcfc/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0e4f79ddc8a675e4ceede6fd99a69b7f24362c40ca650fd1cfd34adb614521e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3aa4e8592435481dce2a3c42568a966941cb471c877cd9c490dd83d1ca6f2ea/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:145a835ff7994fe MacAddress:96:d1:84:ca:fc:cc Speed:10000 Mtu:8901} {Name:2280d4464568344 MacAddress:fa:83:21:13:e7:7f Speed:10000 Mtu:8901} {Name:2edb5feaff3c677 MacAddress:e2:0a:54:66:b7:cc Speed:10000 Mtu:8901} {Name:59b05ffd230698b MacAddress:e2:10:cc:37:15:80 Speed:10000 Mtu:8901} {Name:bd7fe9697c5383a MacAddress:1a:3f:7d:eb:50:33 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:9d:33:95 Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:ba:b5:80:40:a9 Speed:0 Mtu:8901} {Name:c76f53a27f3c51f MacAddress:3e:1d:ec:88:40:68 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:9d:33:95 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:a7:b6:da:15:37 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:6b:29:8b:c2:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 01:14:59.981423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981255 4552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 01:14:59.981423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981314 4552 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 01:14:59.981810 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981782 4552 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 01:14:59.981957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981812 4552 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-1.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 01:14:59.982003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981963 4552 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 01:14:59.982003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981971 4552 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 01:14:59.982003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981984 4552 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:59.982003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.981997 4552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:59.982103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982092 4552 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 01:14:59.982332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982322 4552 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 01:14:59.982470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982449 4552 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 01:14:59.982583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982494 4552 kubelet.go:493] "Attempting to sync node with API server" Nov 02 01:14:59.982583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982511 4552 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 01:14:59.982583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982528 4552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 01:14:59.982583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982537 4552 kubelet.go:405] "Adding apiserver pod source" Nov 02 01:14:59.982583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.982547 4552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 01:14:59.983974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.983955 4552 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 01:14:59.983974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.983973 4552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:59.984746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.984712 4552 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 01:14:59.989293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989276 4552 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 01:14:59.989293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989293 4552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:59.989430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989418 4552 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 01:14:59.989735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989720 4552 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 01:14:59.989788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989777 4552 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 01:14:59.989904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989894 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 01:14:59.989933 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989913 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 01:14:59.989933 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989922 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 01:14:59.989933 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989932 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 01:14:59.990007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989943 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 01:14:59.990007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989951 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 01:14:59.990007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989960 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 01:14:59.990007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989968 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 01:14:59.990007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989978 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 01:14:59.990007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.989988 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 01:14:59.990188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.990008 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 01:14:59.990188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.990022 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 01:14:59.990188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.990058 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 01:14:59.990188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.990068 4552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 01:14:59.994453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.994438 4552 server.go:1263] "Started kubelet" Nov 02 01:14:59.994540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.994514 4552 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 01:14:59.996917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.994687 4552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 01:14:59.996917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.994758 4552 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 01:14:59.995041 ip-10-0-137-1 systemd[1]: Started Kubernetes Kubelet. Nov 02 01:14:59.997423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:14:59.997403 4552 server.go:317] "Adding debug handlers to kubelet server" Nov 02 01:15:00.000485 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.000450 4552 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:15:00.003388 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003356 4552 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:15:00.003495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003410 4552 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 01:15:00.003495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003438 4552 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003605 4552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003698 4552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:20 +0000 UTC" deadline="2028-07-17 13:43:48.015920796 +0000 UTC" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003733 4552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14964h28m48.012191498s" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003775 4552 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003790 4552 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.003910 4552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.004652 4552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.004669 4552 factory.go:55] Registering systemd factory Nov 02 01:15:00.004768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.004679 4552 factory.go:223] Registration of the systemd container factory successfully Nov 02 01:15:00.005297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.004971 4552 factory.go:153] Registering CRI-O factory Nov 02 01:15:00.005297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.004986 4552 factory.go:223] Registration of the crio container factory successfully Nov 02 01:15:00.005297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.005007 4552 factory.go:103] Registering Raw factory Nov 02 01:15:00.005297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.005023 4552 manager.go:1199] Started watching for new ooms in manager Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006610 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4809d979-ef70-435c-b1e5-e403fa49988d" volumeName="kubernetes.io/secret/4809d979-ef70-435c-b1e5-e403fa49988d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006899 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="622df053-819a-4cbb-9ab8-bb507756ac96" volumeName="kubernetes.io/secret/622df053-819a-4cbb-9ab8-bb507756ac96-cert" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006919 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb712b5a-44f2-4a7b-b3da-7bf1e9253da4" volumeName="kubernetes.io/projected/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-kube-api-access-dt9j5" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006935 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="040c33cc-4366-4e4a-8d93-3f35a1bd5cb8" volumeName="kubernetes.io/projected/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-kube-api-access-8xx9j" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006954 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15670b0f-869e-49bf-9da0-9bf6ea903f2c" volumeName="kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006966 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15670b0f-869e-49bf-9da0-9bf6ea903f2c" volumeName="kubernetes.io/projected/15670b0f-869e-49bf-9da0-9bf6ea903f2c-kube-api-access-mv252" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006983 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d617c8-7375-4b58-93b2-7eb39c4bb8c0" volumeName="kubernetes.io/empty-dir/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-tmp-dir" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.006995 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d617c8-7375-4b58-93b2-7eb39c4bb8c0" volumeName="kubernetes.io/projected/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-kube-api-access-f7rdt" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007034 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4809d979-ef70-435c-b1e5-e403fa49988d" volumeName="kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-config" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007047 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6ca2c0e-a09c-4c72-bb67-c595dad9050c" volumeName="kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007063 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a74bd44-3615-4c98-8f3b-4ba10ff959af" volumeName="kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007079 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4809d979-ef70-435c-b1e5-e403fa49988d" volumeName="kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-env-overrides" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007093 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cfdec63-7004-4a85-91dd-6a7a6db01761" volumeName="kubernetes.io/projected/6cfdec63-7004-4a85-91dd-6a7a6db01761-kube-api-access-zsd8x" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007144 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="779489ef-236e-4998-a739-964989ab5d07" volumeName="kubernetes.io/projected/779489ef-236e-4998-a739-964989ab5d07-kube-api-access-gk4lp" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007163 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5aaf1d9-2a69-4a0f-9503-2217191d7131" volumeName="kubernetes.io/projected/d5aaf1d9-2a69-4a0f-9503-2217191d7131-kube-api-access-46rsb" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007180 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb712b5a-44f2-4a7b-b3da-7bf1e9253da4" volumeName="kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-tmp" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007192 4552 manager.go:319] Starting recovery of all containers Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007194 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a74bd44-3615-4c98-8f3b-4ba10ff959af" volumeName="kubernetes.io/projected/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-api-access-wk45f" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007261 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa" volumeName="kubernetes.io/configmap/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-konnectivity-ca" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007295 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c393d2-418a-405e-b9aa-de61625fc694" volumeName="kubernetes.io/secret/58c393d2-418a-405e-b9aa-de61625fc694-signing-key" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007310 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="622df053-819a-4cbb-9ab8-bb507756ac96" volumeName="kubernetes.io/projected/622df053-819a-4cbb-9ab8-bb507756ac96-kube-api-access-sch5m" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007326 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cfdec63-7004-4a85-91dd-6a7a6db01761" volumeName="kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-cni-binary-copy" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007346 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844ca99b-cdd2-4659-bd4d-839c69e15f46" volumeName="kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007359 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="040c33cc-4366-4e4a-8d93-3f35a1bd5cb8" volumeName="kubernetes.io/configmap/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-serviceca" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007380 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15670b0f-869e-49bf-9da0-9bf6ea903f2c" volumeName="kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-binary-copy" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007395 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4809d979-ef70-435c-b1e5-e403fa49988d" volumeName="kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-script-lib" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007414 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4809d979-ef70-435c-b1e5-e403fa49988d" volumeName="kubernetes.io/projected/4809d979-ef70-435c-b1e5-e403fa49988d-kube-api-access-l72cd" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007426 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d635055-2cef-423b-af9b-9dba9996be9e" volumeName="kubernetes.io/projected/5d635055-2cef-423b-af9b-9dba9996be9e-kube-api-access-fhd8h" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007440 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb712b5a-44f2-4a7b-b3da-7bf1e9253da4" volumeName="kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-tuned" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007496 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15670b0f-869e-49bf-9da0-9bf6ea903f2c" volumeName="kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007517 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa" volumeName="kubernetes.io/secret/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-agent-certs" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007530 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4cf42-f58e-442c-857c-1b7c87152cd7" volumeName="kubernetes.io/configmap/34e4cf42-f58e-442c-857c-1b7c87152cd7-config-volume" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007543 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cfdec63-7004-4a85-91dd-6a7a6db01761" volumeName="kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-daemon-config" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007561 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844ca99b-cdd2-4659-bd4d-839c69e15f46" volumeName="kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007591 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc2693cb-eac1-4cd0-b76c-db35732e9635" volumeName="kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007608 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4cf42-f58e-442c-857c-1b7c87152cd7" volumeName="kubernetes.io/empty-dir/34e4cf42-f58e-442c-857c-1b7c87152cd7-tmp-dir" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007625 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c393d2-418a-405e-b9aa-de61625fc694" volumeName="kubernetes.io/configmap/58c393d2-418a-405e-b9aa-de61625fc694-signing-cabundle" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007638 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844ca99b-cdd2-4659-bd4d-839c69e15f46" volumeName="kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007678 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844ca99b-cdd2-4659-bd4d-839c69e15f46" volumeName="kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007690 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5aaf1d9-2a69-4a0f-9503-2217191d7131" volumeName="kubernetes.io/configmap/d5aaf1d9-2a69-4a0f-9503-2217191d7131-iptables-alerter-script" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007707 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a74bd44-3615-4c98-8f3b-4ba10ff959af" volumeName="kubernetes.io/empty-dir/0a74bd44-3615-4c98-8f3b-4ba10ff959af-data-volume" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007719 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4cf42-f58e-442c-857c-1b7c87152cd7" volumeName="kubernetes.io/projected/34e4cf42-f58e-442c-857c-1b7c87152cd7-kube-api-access-z85xq" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007734 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f40d17-3c79-4d12-9f72-58c903869221" volumeName="kubernetes.io/projected/55f40d17-3c79-4d12-9f72-58c903869221-kube-api-access-cz8qd" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007752 4552 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c393d2-418a-405e-b9aa-de61625fc694" volumeName="kubernetes.io/projected/58c393d2-418a-405e-b9aa-de61625fc694-kube-api-access-82qwq" seLinuxMountContext="" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007764 4552 reconstruct.go:97] "Volume reconstruction finished" Nov 02 01:15:00.007950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.007774 4552 reconciler.go:29] "Reconciler: start to sync state" Nov 02 01:15:00.015143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.011016 4552 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:15:00.015143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.011978 4552 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 01:15:00.025262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.025230 4552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 01:15:00.035915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.035887 4552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 01:15:00.035915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.035912 4552 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 01:15:00.036190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.035936 4552 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 01:15:00.036190 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:00.035972 4552 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 01:15:00.039879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.039849 4552 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:15:00.054364 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.054325 4552 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="c0ccb92c8774bfe9124cacaf7b2f30a713ecc8ed272fc2f60ab6d33bb23c88a6" exitCode=0 Nov 02 01:15:00.054364 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.054363 4552 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="9019153bf44540db037e985b3d7014b066d92d675f14ca4bd47177dbaa4d8741" exitCode=0 Nov 02 01:15:00.054576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.054377 4552 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="947e42a90c65ab547f8a5a337a7d20a5d28892e627d7674725d37005b91b0ce8" exitCode=0 Nov 02 01:15:00.054576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.054389 4552 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="6f0785c03a494ea2a78a3c20dcba0402543d023dcfe6f6d4523b039685b4462c" exitCode=0 Nov 02 01:15:00.054576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.054397 4552 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="55897de2bc81ead7334eaeca9437dcabd821daaeebc893f7262aea9e5304eb3c" exitCode=0 Nov 02 01:15:00.054576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.054409 4552 generic.go:356] "Generic (PLEG): container finished" podID="15670b0f-869e-49bf-9da0-9bf6ea903f2c" containerID="0a8704a1d63aacb48c51b635da24e7222ca8b13431df3775e9a55c8b28f47d47" exitCode=0 Nov 02 01:15:00.076878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.076844 4552 generic.go:356] "Generic (PLEG): container finished" podID="990ecd6a432f4311162cd85719f32ab0" containerID="6bd4ed71d5480a82bfc556b9487a005ecf88448cc9f53cacd90a1e571779d6f9" exitCode=0 Nov 02 01:15:00.084449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.084378 4552 manager.go:324] Recovery completed Nov 02 01:15:00.100270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100242 4552 cpu_manager.go:226] "Starting" policy="none" Nov 02 01:15:00.100270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100264 4552 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 01:15:00.100402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100287 4552 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 01:15:00.100463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100445 4552 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 01:15:00.100511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100466 4552 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 01:15:00.100511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100491 4552 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 01:15:00.100511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100499 4552 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 01:15:00.100511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100506 4552 policy_none.go:50] "Start" Nov 02 01:15:00.100618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100518 4552 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 01:15:00.100618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100532 4552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 01:15:00.100682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100672 4552 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 01:15:00.100714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100683 4552 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 01:15:00.100714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.100690 4552 policy_none.go:44] "Start" Nov 02 01:15:00.106125 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.106079 4552 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 01:15:00.108941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.108928 4552 manager.go:341] "Starting Device Plugin manager" Nov 02 01:15:00.108998 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:00.108956 4552 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 01:15:00.108998 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.108967 4552 server.go:87] "Starting device plugin registration server" Nov 02 01:15:00.109771 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.109747 4552 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 01:15:00.109867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.109772 4552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 01:15:00.109915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.109874 4552 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 01:15:00.110000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.109968 4552 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 01:15:00.110064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.110010 4552 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 01:15:00.110064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.110016 4552 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 01:15:00.110163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.110074 4552 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:15:00.109998023Z","UUID":"ccd2d12c-3762-49df-be96-70fdcdeb9139","Handler":null,"Name":"","Endpoint":""} Nov 02 01:15:00.111945 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:00.111921 4552 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 01:15:00.112446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.112411 4552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 01:15:00.112537 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.112455 4552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 01:15:00.136421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.136378 4552 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal"] Nov 02 01:15:00.136966 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.136927 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"0100e2035cafa00853427683c7ebd01f","Type":"ContainerStarted","Data":"9cc84f9aa0d1cca5f29f1026a36fd999db66e337581776dfdbcb479e3a26e56d"} Nov 02 01:15:00.137023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.136980 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"0100e2035cafa00853427683c7ebd01f","Type":"ContainerStarted","Data":"2f4f27a44b3a61f8562009aaa8e51197a92ddc47934cf0f069029426c6698e1b"} Nov 02 01:15:00.137023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.136998 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"990ecd6a432f4311162cd85719f32ab0","Type":"ContainerStarted","Data":"dd2af87834ad561b3157d2336008243785304d2ab0bb832a003cd8ac7fef7d2b"} Nov 02 01:15:00.137023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.137017 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"990ecd6a432f4311162cd85719f32ab0","Type":"ContainerDied","Data":"6bd4ed71d5480a82bfc556b9487a005ecf88448cc9f53cacd90a1e571779d6f9"} Nov 02 01:15:00.137140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.137035 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" event={"ID":"990ecd6a432f4311162cd85719f32ab0","Type":"ContainerStarted","Data":"fd65fa4943c78268881318ca4d1f03a2d27f84c52ed730558e4fcfb366d7bb64"} Nov 02 01:15:00.137140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.137000 4552 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.137206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.137151 4552 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.149439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.149412 4552 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:15:00.149625 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:00.149472 4552 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.149625 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.149485 4552 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:15:00.149625 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:00.149524 4552 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.210839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.210803 4552 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:15:00.212219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212201 4552 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:15:00.212304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212226 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.212304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212256 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.212304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212281 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0100e2035cafa00853427683c7ebd01f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"0100e2035cafa00853427683c7ebd01f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.212304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212232 4552 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:15:00.212672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212653 4552 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:15:00.212766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.212739 4552 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.227781 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.227755 4552 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.227898 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.227812 4552 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.312671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.312632 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.312671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.312676 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.312973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.312701 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0100e2035cafa00853427683c7ebd01f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"0100e2035cafa00853427683c7ebd01f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.312973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.312747 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0100e2035cafa00853427683c7ebd01f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"0100e2035cafa00853427683c7ebd01f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.312973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.312748 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.312973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.312751 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990ecd6a432f4311162cd85719f32ab0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" (UID: \"990ecd6a432f4311162cd85719f32ab0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:00.983410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.983377 4552 apiserver.go:52] "Watching apiserver" Nov 02 01:15:00.991941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.991918 4552 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 01:15:00.992415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.992396 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-8m9jp","openshift-service-ca/service-ca-5fffc97bdf-t9rts","openshift-insights/periodic-gathering-74jqn-trlw4","openshift-ingress-canary/ingress-canary-lmft6","openshift-network-operator/iptables-alerter-x82sf","openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh","openshift-ovn-kubernetes/ovnkube-node-mcfgw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal","openshift-dns/dns-default-rd27f","openshift-image-registry/node-ca-zttdn","openshift-multus/network-metrics-daemon-wx6pd","kube-system/global-pull-secret-syncer-tqwk8","kube-system/konnectivity-agent-qjmzq","openshift-cluster-node-tuning-operator/tuned-kgnc2","openshift-multus/multus-j5k6h","openshift-network-diagnostics/network-check-target-4x747","openshift-dns/node-resolver-vts4p","openshift-insights/insights-runtime-extractor-5cb8t"] Nov 02 01:15:00.993305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.993272 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:00.997048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:00.997027 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.000589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000568 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 01:15:01.000778 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000760 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 01:15:01.000844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000777 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.000844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000809 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 01:15:01.000844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000816 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.000844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000822 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000852 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000865 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000906 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wfrhp\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000914 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000919 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000936 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000965 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000970 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67z2p\"" Nov 02 01:15:01.001017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.000974 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.001935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.001913 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 01:15:01.002010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.001990 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-87cgr\"" Nov 02 01:15:01.003189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.003171 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.003615 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.003599 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.003707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.003695 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 01:15:01.003768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.003754 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 01:15:01.004158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004143 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 01:15:01.004246 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004187 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 01:15:01.004294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004277 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pl9l\"" Nov 02 01:15:01.004387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004370 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tmdq2\"" Nov 02 01:15:01.004444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004432 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 01:15:01.004529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004496 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.004529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004512 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 01:15:01.005018 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.004998 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6rtc\"" Nov 02 01:15:01.005149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005032 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.005149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005063 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 01:15:01.005149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005080 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5kn9m\"" Nov 02 01:15:01.005149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005103 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005196 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75lmf\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005208 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005224 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005270 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85fmt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005285 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005297 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005319 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005366 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005375 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005400 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005271 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlhpz\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005415 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7fmpb\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005427 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 01:15:01.005437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005452 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rcvqj\"" Nov 02 01:15:01.005942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005459 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 01:15:01.005942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005005 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8rk5x\"" Nov 02 01:15:01.005942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005501 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g7hgp\"" Nov 02 01:15:01.005942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.005504 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.006229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006175 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ck74h\"" Nov 02 01:15:01.006229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006184 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.006366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006234 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.006366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006334 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006384 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006400 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006412 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006414 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006427 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006462 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zb852\"" Nov 02 01:15:01.006516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006511 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 01:15:01.006744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.006643 4552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 01:15:01.007726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.007709 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5n6sn\"" Nov 02 01:15:01.010668 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.010647 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 01:15:01.010766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.010730 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 01:15:01.010766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.010750 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hdjbk\"" Nov 02 01:15:01.010858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.010843 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 01:15:01.015961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.015939 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-tmp-dir\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.016056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.015973 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rdt\" (UniqueName: \"kubernetes.io/projected/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-kube-api-access-f7rdt\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.016056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016000 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.016187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016058 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-kubelet\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.016187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016081 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-device-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.016187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016101 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.016187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016101 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-tmp-dir\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.016187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016162 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-hosts-file\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016201 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-log-socket\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016236 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-systemd-units\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016265 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-var-lib-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016296 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016357 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85xq\" (UniqueName: \"kubernetes.io/projected/34e4cf42-f58e-442c-857c-1b7c87152cd7-kube-api-access-z85xq\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016382 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rsb\" (UniqueName: \"kubernetes.io/projected/d5aaf1d9-2a69-4a0f-9503-2217191d7131-kube-api-access-46rsb\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016402 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.016458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016447 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58c393d2-418a-405e-b9aa-de61625fc694-signing-cabundle\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016516 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016575 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016608 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-multus-certs\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016635 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-registration-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016665 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016689 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-var-lib-kubelet\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016692 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58c393d2-418a-405e-b9aa-de61625fc694-signing-cabundle\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:15:01.016829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016804 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk45f\" (UniqueName: \"kubernetes.io/projected/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-api-access-wk45f\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016850 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cnibin\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016871 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016875 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-slash\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016899 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-host\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016951 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qwq\" (UniqueName: \"kubernetes.io/projected/58c393d2-418a-405e-b9aa-de61625fc694-kube-api-access-82qwq\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.016990 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-agent-certs\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017014 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/622df053-819a-4cbb-9ab8-bb507756ac96-cert\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017030 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-cni-binary-copy\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017061 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-kubelet-config\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017087 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv252\" (UniqueName: \"kubernetes.io/projected/15670b0f-869e-49bf-9da0-9bf6ea903f2c-kube-api-access-mv252\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017133 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.017162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017158 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72cd\" (UniqueName: \"kubernetes.io/projected/4809d979-ef70-435c-b1e5-e403fa49988d-kube-api-access-l72cd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017213 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4cf42-f58e-442c-857c-1b7c87152cd7-tmp-dir\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017239 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017265 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a74bd44-3615-4c98-8f3b-4ba10ff959af-crio-socket\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017300 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/622df053-819a-4cbb-9ab8-bb507756ac96-cert\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017291 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017324 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-cni-binary-copy\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017336 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-systemd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017401 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4cf42-f58e-442c-857c-1b7c87152cd7-tmp-dir\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017450 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4lp\" (UniqueName: \"kubernetes.io/projected/779489ef-236e-4998-a739-964989ab5d07-kube-api-access-gk4lp\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017476 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-run\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017511 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-host\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017526 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-tuned\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017523 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-agent-certs\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017569 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sch5m\" (UniqueName: \"kubernetes.io/projected/622df053-819a-4cbb-9ab8-bb507756ac96-kube-api-access-sch5m\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017590 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-kubelet\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017600 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-tuned\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.017677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017618 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-etc-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017684 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017739 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc2693cb-eac1-4cd0-b76c-db35732e9635-original-pull-secret\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017738 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-script-lib\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017841 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-socket-dir-parent\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017880 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017910 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-sys\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017908 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-script-lib\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.017939 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018053 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018083 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-cnibin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018109 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd8x\" (UniqueName: \"kubernetes.io/projected/6cfdec63-7004-4a85-91dd-6a7a6db01761-kube-api-access-zsd8x\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018154 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018183 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58c393d2-418a-405e-b9aa-de61625fc694-signing-key\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018196 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018207 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-dbus\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018233 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018257 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018291 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.018336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018297 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4809d979-ef70-435c-b1e5-e403fa49988d-ovn-node-metrics-cert\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018371 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-socket-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018389 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58c393d2-418a-405e-b9aa-de61625fc694-signing-key\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018391 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018409 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018447 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-conf\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018451 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4809d979-ef70-435c-b1e5-e403fa49988d-ovn-node-metrics-cert\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018474 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-tmp\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018483 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018500 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9j5\" (UniqueName: \"kubernetes.io/projected/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-kube-api-access-dt9j5\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018527 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-netd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018548 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-tmp\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018552 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-k8s-cni-cncf-io\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018591 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhd8h\" (UniqueName: \"kubernetes.io/projected/5d635055-2cef-423b-af9b-9dba9996be9e-kube-api-access-fhd8h\") pod \"migrator-66df5d496b-nwfrh\" (UID: \"5d635055-2cef-423b-af9b-9dba9996be9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018621 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018648 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-modprobe-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018689 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-kubernetes\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018727 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-systemd\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018775 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-netns\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018800 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-netns\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018827 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-daemon-config\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.018946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018869 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xx9j\" (UniqueName: \"kubernetes.io/projected/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-kube-api-access-8xx9j\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018978 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5aaf1d9-2a69-4a0f-9503-2217191d7131-iptables-alerter-script\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.018984 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-daemon-config\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019109 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5aaf1d9-2a69-4a0f-9503-2217191d7131-host-slash\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019176 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysconfig\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019216 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019269 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5aaf1d9-2a69-4a0f-9503-2217191d7131-iptables-alerter-script\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019271 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-bin\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019317 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-env-overrides\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019350 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-conf-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019391 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-sys-fs\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019426 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34e4cf42-f58e-442c-857c-1b7c87152cd7-config-volume\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019468 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz8qd\" (UniqueName: \"kubernetes.io/projected/55f40d17-3c79-4d12-9f72-58c903869221-kube-api-access-cz8qd\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019478 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-env-overrides\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019493 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-lib-modules\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019515 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-system-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019541 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019558 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-serviceca\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019611 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019641 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-ovn\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019660 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34e4cf42-f58e-442c-857c-1b7c87152cd7-config-volume\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.019708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019680 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-config\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019765 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-bin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019815 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-serviceca\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019867 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-hostroot\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019908 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-etc-kubernetes\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019919 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4809d979-ef70-435c-b1e5-e403fa49988d-ovnkube-config\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019926 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019962 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-node-log\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.019993 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-multus\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.020035 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.020060 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a74bd44-3615-4c98-8f3b-4ba10ff959af-data-volume\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.020153 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-konnectivity-ca\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.020166 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a74bd44-3615-4c98-8f3b-4ba10ff959af-data-volume\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.020378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.020299 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa-konnectivity-ca\") pod \"konnectivity-agent-qjmzq\" (UID: \"31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa\") " pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:15:01.025813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.025796 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rdt\" (UniqueName: \"kubernetes.io/projected/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-kube-api-access-f7rdt\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.025999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.025979 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85xq\" (UniqueName: \"kubernetes.io/projected/34e4cf42-f58e-442c-857c-1b7c87152cd7-kube-api-access-z85xq\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.026688 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.026672 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rsb\" (UniqueName: \"kubernetes.io/projected/d5aaf1d9-2a69-4a0f-9503-2217191d7131-kube-api-access-46rsb\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.026921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.026906 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk45f\" (UniqueName: \"kubernetes.io/projected/0a74bd44-3615-4c98-8f3b-4ba10ff959af-kube-api-access-wk45f\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.027539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.027524 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qwq\" (UniqueName: \"kubernetes.io/projected/58c393d2-418a-405e-b9aa-de61625fc694-kube-api-access-82qwq\") pod \"service-ca-5fffc97bdf-t9rts\" (UID: \"58c393d2-418a-405e-b9aa-de61625fc694\") " pod="openshift-service-ca/service-ca-5fffc97bdf-t9rts" Nov 02 01:15:01.028662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.028642 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd8x\" (UniqueName: \"kubernetes.io/projected/6cfdec63-7004-4a85-91dd-6a7a6db01761-kube-api-access-zsd8x\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.029221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.029197 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sch5m\" (UniqueName: \"kubernetes.io/projected/622df053-819a-4cbb-9ab8-bb507756ac96-kube-api-access-sch5m\") pod \"ingress-canary-lmft6\" (UID: \"622df053-819a-4cbb-9ab8-bb507756ac96\") " pod="openshift-ingress-canary/ingress-canary-lmft6" Nov 02 01:15:01.029491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.029466 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9j5\" (UniqueName: \"kubernetes.io/projected/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-kube-api-access-dt9j5\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.029714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.029692 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72cd\" (UniqueName: \"kubernetes.io/projected/4809d979-ef70-435c-b1e5-e403fa49988d-kube-api-access-l72cd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.031169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.031144 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/f6ca2c0e-a09c-4c72-bb67-c595dad9050c-kube-api-access-fr7fg\") pod \"network-check-target-4x747\" (UID: \"f6ca2c0e-a09c-4c72-bb67-c595dad9050c\") " pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:15:01.031493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.031470 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz8qd\" (UniqueName: \"kubernetes.io/projected/55f40d17-3c79-4d12-9f72-58c903869221-kube-api-access-cz8qd\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:01.031867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.031848 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhd8h\" (UniqueName: \"kubernetes.io/projected/5d635055-2cef-423b-af9b-9dba9996be9e-kube-api-access-fhd8h\") pod \"migrator-66df5d496b-nwfrh\" (UID: \"5d635055-2cef-423b-af9b-9dba9996be9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nwfrh" Nov 02 01:15:01.032452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.032240 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") pod \"periodic-gathering-74jqn-trlw4\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:15:01.032784 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.032764 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4lp\" (UniqueName: \"kubernetes.io/projected/779489ef-236e-4998-a739-964989ab5d07-kube-api-access-gk4lp\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.032844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.032765 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xx9j\" (UniqueName: \"kubernetes.io/projected/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-kube-api-access-8xx9j\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.033738 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.033720 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv252\" (UniqueName: \"kubernetes.io/projected/15670b0f-869e-49bf-9da0-9bf6ea903f2c-kube-api-access-mv252\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.079358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.079331 4552 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:01.079483 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.079400 4552 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:01.091230 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.091206 4552 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:15:01.091330 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.091252 4552 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:01.091859 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.091845 4552 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:15:01.091895 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.091885 4552 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-east-2.compute.internal" Nov 02 01:15:01.120593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120564 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysconfig\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.120593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120591 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120608 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-bin\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120624 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-conf-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120638 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-sys-fs\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120653 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-lib-modules\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120677 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-system-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120697 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120708 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-conf-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120730 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-sys-fs\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120709 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysconfig\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.120757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120757 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-system-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120761 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-cni-dir\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120757 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120734 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-bin\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120816 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-ovn\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.120765 4552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.120841 4552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120840 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-lib-modules\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120880 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-bin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120849 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-bin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120914 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-ovn\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.120951 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:01.620912569 +0000 UTC m=+1.726701482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.120974 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:15:01.620960421 +0000 UTC m=+1.726749313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.120989 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-hostroot\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121006 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-etc-kubernetes\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121021 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121036 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-node-log\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121051 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-multus\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121070 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121088 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-etc-kubernetes\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121095 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-node-log\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121093 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-kubelet\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121111 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-hostroot\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121153 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-cni-multus\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121155 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-var-lib-kubelet\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121171 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121177 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-device-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121203 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121220 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-device-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.121239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121236 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-hosts-file\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121276 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-log-socket\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121292 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-systemd-units\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121314 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-systemd-units\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121339 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-log-socket\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121347 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30d617c8-7375-4b58-93b2-7eb39c4bb8c0-hosts-file\") pod \"node-resolver-vts4p\" (UID: \"30d617c8-7375-4b58-93b2-7eb39c4bb8c0\") " pod="openshift-dns/node-resolver-vts4p" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121370 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-var-lib-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121389 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121404 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121420 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121426 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-var-lib-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121436 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-multus-certs\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121461 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121465 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-multus-certs\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121479 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121489 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-registration-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121512 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-var-lib-kubelet\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121518 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121537 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cnibin\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121552 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-var-lib-kubelet\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121559 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-slash\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121580 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-slash\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121583 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-host\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121550 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-registration-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121614 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-kubelet-config\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121621 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/040c33cc-4366-4e4a-8d93-3f35a1bd5cb8-host\") pod \"node-ca-zttdn\" (UID: \"040c33cc-4366-4e4a-8d93-3f35a1bd5cb8\") " pod="openshift-image-registry/node-ca-zttdn" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121578 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15670b0f-869e-49bf-9da0-9bf6ea903f2c-cnibin\") pod \"multus-additional-cni-plugins-8m9jp\" (UID: \"15670b0f-869e-49bf-9da0-9bf6ea903f2c\") " pod="openshift-multus/multus-additional-cni-plugins-8m9jp" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121639 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121672 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-kubelet-config\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121670 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121692 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a74bd44-3615-4c98-8f3b-4ba10ff959af-crio-socket\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121710 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-systemd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121729 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-run\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121741 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-host\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121755 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-kubelet\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121775 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-run-systemd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121777 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-etc-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121804 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-socket-dir-parent\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121808 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-etc-openvswitch\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121817 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-run\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121823 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a74bd44-3615-4c98-8f3b-4ba10ff959af-crio-socket\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121832 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-host\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121822 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121843 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-kubelet\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121857 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-sys\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121876 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-cnibin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121884 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-multus-socket-dir-parent\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121902 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-dbus\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121859 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121923 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-sys\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121927 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121940 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-cnibin\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121960 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121963 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-socket-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121989 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.121966 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc2693cb-eac1-4cd0-b76c-db35732e9635-dbus\") pod \"global-pull-secret-syncer-tqwk8\" (UID: \"dc2693cb-eac1-4cd0-b76c-db35732e9635\") " pod="kube-system/global-pull-secret-syncer-tqwk8" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122005 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-conf\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122027 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-netd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122036 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779489ef-236e-4998-a739-964989ab5d07-socket-dir\") pod \"aws-ebs-csi-driver-node-86cb5\" (UID: \"779489ef-236e-4998-a739-964989ab5d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86cb5" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122043 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-k8s-cni-cncf-io\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122058 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-modprobe-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122062 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-cni-netd\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122072 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-kubernetes\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122083 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-k8s-cni-cncf-io\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122091 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-sysctl-conf\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122094 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-systemd\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122141 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-systemd\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.122143 4552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122146 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-netns\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122158 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-kubernetes\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122169 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb712b5a-44f2-4a7b-b3da-7bf1e9253da4-etc-modprobe-d\") pod \"tuned-kgnc2\" (UID: \"eb712b5a-44f2-4a7b-b3da-7bf1e9253da4\") " pod="openshift-cluster-node-tuning-operator/tuned-kgnc2" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122176 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4809d979-ef70-435c-b1e5-e403fa49988d-host-run-netns\") pod \"ovnkube-node-mcfgw\" (UID: \"4809d979-ef70-435c-b1e5-e403fa49988d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122183 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-netns\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.122200 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:01.622189581 +0000 UTC m=+1.727978502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : secret "metrics-daemon-secret" not found Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122222 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cfdec63-7004-4a85-91dd-6a7a6db01761-host-run-netns\") pod \"multus-j5k6h\" (UID: \"6cfdec63-7004-4a85-91dd-6a7a6db01761\") " pod="openshift-multus/multus-j5k6h" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122226 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5aaf1d9-2a69-4a0f-9503-2217191d7131-host-slash\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.122342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.122251 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5aaf1d9-2a69-4a0f-9503-2217191d7131-host-slash\") pod \"iptables-alerter-x82sf\" (UID: \"d5aaf1d9-2a69-4a0f-9503-2217191d7131\") " pod="openshift-network-operator/iptables-alerter-x82sf" Nov 02 01:15:01.626098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.626061 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:01.626285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.626105 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:01.626285 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.626217 4552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:15:01.626285 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.626226 4552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:15:01.626285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.626261 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:01.626285 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.626275 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:15:02.626261765 +0000 UTC m=+2.732050657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:15:01.626444 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.626290 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:02.626283741 +0000 UTC m=+2.732072633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : secret "metrics-daemon-secret" not found Nov 02 01:15:01.626444 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.626336 4552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:15:01.626444 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:01.626376 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:02.62636426 +0000 UTC m=+2.732153156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:15:01.950360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:01.950284 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:15:02.632510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:02.632478 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:02.632510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:02.632514 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:02.632535 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:02.632632 4552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:02.632640 4552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:02.632682 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls podName:34e4cf42-f58e-442c-857c-1b7c87152cd7 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:04.632669346 +0000 UTC m=+4.738458251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls") pod "dns-default-rd27f" (UID: "34e4cf42-f58e-442c-857c-1b7c87152cd7") : secret "dns-default-metrics-tls" not found Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:02.632710 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:15:04.632697199 +0000 UTC m=+4.738486115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:02.632640 4552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:15:02.632968 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:02.632759 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:04.632748933 +0000 UTC m=+4.738537840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : secret "metrics-daemon-secret" not found Nov 02 01:15:03.108774 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:03.108747 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:03.980050 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:03.980020 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vts4p_30d617c8-7375-4b58-93b2-7eb39c4bb8c0/dns-node-resolver/0.log" Nov 02 01:15:04.646779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.646730 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:04.646779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.646790 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:04.646992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.646812 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:04.646992 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:04.646885 4552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:15:04.646992 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:04.646932 4552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:15:04.646992 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:04.646987 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs podName:55f40d17-3c79-4d12-9f72-58c903869221 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:08.646965272 +0000 UTC m=+8.752754188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs") pod "network-metrics-daemon-wx6pd" (UID: "55f40d17-3c79-4d12-9f72-58c903869221") : secret "metrics-daemon-secret" not found Nov 02 01:15:04.647150 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:04.647001 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls podName:0a74bd44-3615-4c98-8f3b-4ba10ff959af nodeName:}" failed. No retries permitted until 2026-11-02 01:15:08.646995357 +0000 UTC m=+8.752784249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5cb8t" (UID: "0a74bd44-3615-4c98-8f3b-4ba10ff959af") : secret "insights-runtime-extractor-tls" not found Nov 02 01:15:04.647202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.647184 4552 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 01:15:04.649312 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.649294 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34e4cf42-f58e-442c-857c-1b7c87152cd7-metrics-tls\") pod \"dns-default-rd27f\" (UID: \"34e4cf42-f58e-442c-857c-1b7c87152cd7\") " pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:04.897815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.897721 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:04.980173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:04.980148 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zttdn_040c33cc-4366-4e4a-8d93-3f35a1bd5cb8/node-ca/0.log" Nov 02 01:15:05.169067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:05.168978 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rd27f"] Nov 02 01:15:05.172944 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:05.172903 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e4cf42_f58e_442c_857c_1b7c87152cd7.slice/crio-be10598f810e42a2077728ec5938ebe659d4708c482e7e0ded2d76ac1e640dcc WatchSource:0}: Error finding container be10598f810e42a2077728ec5938ebe659d4708c482e7e0ded2d76ac1e640dcc: Status 404 returned error can't find the container with id be10598f810e42a2077728ec5938ebe659d4708c482e7e0ded2d76ac1e640dcc Nov 02 01:15:05.175034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:05.175015 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:15:05.380708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:05.380673 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmft6_622df053-819a-4cbb-9ab8-bb507756ac96/serve-healthcheck-canary/0.log" Nov 02 01:15:06.092602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:06.092561 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rd27f" event={"ID":"34e4cf42-f58e-442c-857c-1b7c87152cd7","Type":"ContainerStarted","Data":"be10598f810e42a2077728ec5938ebe659d4708c482e7e0ded2d76ac1e640dcc"} Nov 02 01:15:06.180439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:06.180409 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nwfrh_5d635055-2cef-423b-af9b-9dba9996be9e/migrator/0.log" Nov 02 01:15:08.104366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.104319 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rd27f" event={"ID":"34e4cf42-f58e-442c-857c-1b7c87152cd7","Type":"ContainerStarted","Data":"c69ed2a7422c3f8a91843ec7cd786a3993224d2fc88389a3fb95d19adf2b07f4"} Nov 02 01:15:08.104790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.104412 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rd27f" event={"ID":"34e4cf42-f58e-442c-857c-1b7c87152cd7","Type":"ContainerStarted","Data":"85e5b045cc7f1b312c981300802e1554075d7272effec189b8c7f45fcdf3378d"} Nov 02 01:15:08.104790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.104472 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:08.680440 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.680409 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:08.680624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.680446 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:08.682927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.682904 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a74bd44-3615-4c98-8f3b-4ba10ff959af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5cb8t\" (UID: \"0a74bd44-3615-4c98-8f3b-4ba10ff959af\") " pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:08.683048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.682943 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f40d17-3c79-4d12-9f72-58c903869221-metrics-certs\") pod \"network-metrics-daemon-wx6pd\" (UID: \"55f40d17-3c79-4d12-9f72-58c903869221\") " pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:08.794058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.794019 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wx6pd" Nov 02 01:15:08.805930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:08.805901 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5cb8t" Nov 02 01:15:09.071358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:09.071321 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wx6pd"] Nov 02 01:15:09.076584 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:09.076558 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f40d17_3c79_4d12_9f72_58c903869221.slice/crio-22a07be4afc92455fd56effe6ab332234f9bc4815d5b195406009b1284896512 WatchSource:0}: Error finding container 22a07be4afc92455fd56effe6ab332234f9bc4815d5b195406009b1284896512: Status 404 returned error can't find the container with id 22a07be4afc92455fd56effe6ab332234f9bc4815d5b195406009b1284896512 Nov 02 01:15:09.093175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:09.092878 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5cb8t"] Nov 02 01:15:09.097335 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:09.097316 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a74bd44_3615_4c98_8f3b_4ba10ff959af.slice/crio-9b43f9f95ba72bde434d2dc87031bbf4494a4453abe14c6d51153299db7f4e45 WatchSource:0}: Error finding container 9b43f9f95ba72bde434d2dc87031bbf4494a4453abe14c6d51153299db7f4e45: Status 404 returned error can't find the container with id 9b43f9f95ba72bde434d2dc87031bbf4494a4453abe14c6d51153299db7f4e45 Nov 02 01:15:09.109033 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:09.109008 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5cb8t" event={"ID":"0a74bd44-3615-4c98-8f3b-4ba10ff959af","Type":"ContainerStarted","Data":"9b43f9f95ba72bde434d2dc87031bbf4494a4453abe14c6d51153299db7f4e45"} Nov 02 01:15:09.110316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:09.110289 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wx6pd" event={"ID":"55f40d17-3c79-4d12-9f72-58c903869221","Type":"ContainerStarted","Data":"22a07be4afc92455fd56effe6ab332234f9bc4815d5b195406009b1284896512"} Nov 02 01:15:09.251585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:09.251556 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:09.251759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:09.251748 4552 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:15:10.116044 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:10.116014 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5cb8t" event={"ID":"0a74bd44-3615-4c98-8f3b-4ba10ff959af","Type":"ContainerStarted","Data":"ad874f75aa6456593ae622ef3def9dfa63712c7df9d4d2aac1e667446230908f"} Nov 02 01:15:11.121640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:11.121603 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wx6pd" event={"ID":"55f40d17-3c79-4d12-9f72-58c903869221","Type":"ContainerStarted","Data":"27fd6a577565bb0149df889127ecce81277abf79becc2460cfc63f379d639d7a"} Nov 02 01:15:11.122191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:11.121698 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wx6pd" event={"ID":"55f40d17-3c79-4d12-9f72-58c903869221","Type":"ContainerStarted","Data":"bd469329e241e6bd502a96ec9ce91c827173ef46d56693b37da765de34ff8c5a"} Nov 02 01:15:11.123248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:11.123226 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5cb8t" event={"ID":"0a74bd44-3615-4c98-8f3b-4ba10ff959af","Type":"ContainerStarted","Data":"539a662268c4efa10f50b36f6420c549023c549cadec6b09258101fa35b2db51"} Nov 02 01:15:12.123271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.123243 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4"] Nov 02 01:15:12.127012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.126992 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-qj4wc"] Nov 02 01:15:12.127161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.127141 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:12.129989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.129968 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.134156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.134103 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sp75l\"" Nov 02 01:15:12.134374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.134360 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 01:15:12.137435 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.137415 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-m98sm\"" Nov 02 01:15:12.138496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.138474 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4"] Nov 02 01:15:12.142156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.141750 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 01:15:12.142156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.141799 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 01:15:12.142156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.142084 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qj4wc"] Nov 02 01:15:12.203985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.203946 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec0430a9-64d9-40ed-94b3-5ad4c9c826b8-tmp\") pod \"downloads-6dd789c8b7-qj4wc\" (UID: \"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8\") " pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.204217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.204093 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrl8\" (UniqueName: \"kubernetes.io/projected/ec0430a9-64d9-40ed-94b3-5ad4c9c826b8-kube-api-access-zjrl8\") pod \"downloads-6dd789c8b7-qj4wc\" (UID: \"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8\") " pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.204217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.204163 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57fd3745-c4ca-4623-9601-48f70b557fc9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cfvc4\" (UID: \"57fd3745-c4ca-4623-9601-48f70b557fc9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:12.305097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.305059 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec0430a9-64d9-40ed-94b3-5ad4c9c826b8-tmp\") pod \"downloads-6dd789c8b7-qj4wc\" (UID: \"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8\") " pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.305378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.305142 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrl8\" (UniqueName: \"kubernetes.io/projected/ec0430a9-64d9-40ed-94b3-5ad4c9c826b8-kube-api-access-zjrl8\") pod \"downloads-6dd789c8b7-qj4wc\" (UID: \"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8\") " pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.305378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.305177 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57fd3745-c4ca-4623-9601-48f70b557fc9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cfvc4\" (UID: \"57fd3745-c4ca-4623-9601-48f70b557fc9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:12.305502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.305483 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec0430a9-64d9-40ed-94b3-5ad4c9c826b8-tmp\") pod \"downloads-6dd789c8b7-qj4wc\" (UID: \"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8\") " pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.307871 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.307843 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57fd3745-c4ca-4623-9601-48f70b557fc9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cfvc4\" (UID: \"57fd3745-c4ca-4623-9601-48f70b557fc9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:12.319784 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.319724 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrl8\" (UniqueName: \"kubernetes.io/projected/ec0430a9-64d9-40ed-94b3-5ad4c9c826b8-kube-api-access-zjrl8\") pod \"downloads-6dd789c8b7-qj4wc\" (UID: \"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8\") " pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.439860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.439829 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:12.447326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.447294 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:12.716316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.716292 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4"] Nov 02 01:15:12.726931 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:12.726903 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qj4wc"] Nov 02 01:15:12.729990 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:12.729965 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0430a9_64d9_40ed_94b3_5ad4c9c826b8.slice/crio-ae6706c04ae6af1848d656a3f3fdff481339455d5cdb2ac83196591009b51a89 WatchSource:0}: Error finding container ae6706c04ae6af1848d656a3f3fdff481339455d5cdb2ac83196591009b51a89: Status 404 returned error can't find the container with id ae6706c04ae6af1848d656a3f3fdff481339455d5cdb2ac83196591009b51a89 Nov 02 01:15:13.132977 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:13.132938 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" event={"ID":"57fd3745-c4ca-4623-9601-48f70b557fc9","Type":"ContainerStarted","Data":"4a00e238ca68c04e0469322f2c0485727404278a13a196a1d743c08f3d6a92f3"} Nov 02 01:15:13.134145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:13.134099 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qj4wc" event={"ID":"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8","Type":"ContainerStarted","Data":"ae6706c04ae6af1848d656a3f3fdff481339455d5cdb2ac83196591009b51a89"} Nov 02 01:15:13.136101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:13.136080 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5cb8t" event={"ID":"0a74bd44-3615-4c98-8f3b-4ba10ff959af","Type":"ContainerStarted","Data":"d56a00ebb5497a3e8d19eeb5aa2811690108f6be83a6decec3981b49ef455fb3"} Nov 02 01:15:14.141543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.141497 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" event={"ID":"57fd3745-c4ca-4623-9601-48f70b557fc9","Type":"ContainerStarted","Data":"db3d08ff5efa223fd52f055cb434ca4920aab5d1cadd662705b8f9f52d1ee9e5"} Nov 02 01:15:14.141987 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.141704 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:14.147503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.147444 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" Nov 02 01:15:14.160645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.160293 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cfvc4" podStartSLOduration=0.979392354 podStartE2EDuration="2.160278278s" totalImagesPullingTime="1.180885924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:14.159983265 +0000 UTC m=+14.265772182" watchObservedRunningTime="2026-11-02 01:15:14.160278278 +0000 UTC m=+14.266067194" Nov 02 01:15:14.163862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.163840 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c6d5d46b6-grhbf"] Nov 02 01:15:14.167166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.167145 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.174711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.174693 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 01:15:14.176995 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.176972 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b7gx2\"" Nov 02 01:15:14.177430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.177406 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 01:15:14.177603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.177454 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 01:15:14.177603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.177519 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 01:15:14.178180 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.178165 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c6d5d46b6-grhbf"] Nov 02 01:15:14.220936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.220903 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.221101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.220948 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.221101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.220983 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lmz\" (UniqueName: \"kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.221101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.221011 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.221101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.221040 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.221101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.221069 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.221385 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.221140 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.274904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.274876 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 01:15:14.322300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322271 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322311 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322333 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322354 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lmz\" (UniqueName: \"kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322373 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322396 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322425 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.322931 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.322908 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.323238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.323215 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.323311 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.323249 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.323411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.323378 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.324851 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.324832 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.325041 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.325019 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.335598 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.335573 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lmz\" (UniqueName: \"kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz\") pod \"console-5c6d5d46b6-grhbf\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.479999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.479968 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:14.769807 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:14.769720 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c6d5d46b6-grhbf"] Nov 02 01:15:14.773445 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:14.773414 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e5a629_74d2_4e37_9867_cb9bd4f9bea9.slice/crio-b1431c09b9b4265150c5abf4ac92e9c3a1065147cf86be58603765e4fc4b6a5f WatchSource:0}: Error finding container b1431c09b9b4265150c5abf4ac92e9c3a1065147cf86be58603765e4fc4b6a5f: Status 404 returned error can't find the container with id b1431c09b9b4265150c5abf4ac92e9c3a1065147cf86be58603765e4fc4b6a5f Nov 02 01:15:15.038772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:15.038689 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcfgw" Nov 02 01:15:15.145812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:15.145752 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6d5d46b6-grhbf" event={"ID":"16e5a629-74d2-4e37-9867-cb9bd4f9bea9","Type":"ContainerStarted","Data":"b1431c09b9b4265150c5abf4ac92e9c3a1065147cf86be58603765e4fc4b6a5f"} Nov 02 01:15:18.113297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:18.113262 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rd27f" Nov 02 01:15:19.163233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.163195 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6d5d46b6-grhbf" event={"ID":"16e5a629-74d2-4e37-9867-cb9bd4f9bea9","Type":"ContainerStarted","Data":"77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8"} Nov 02 01:15:19.186728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.186672 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c6d5d46b6-grhbf" podStartSLOduration=1.7126043260000001 podStartE2EDuration="5.186658398s" totalImagesPullingTime="3.474054072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:19.185227984 +0000 UTC m=+19.291016897" watchObservedRunningTime="2026-11-02 01:15:19.186658398 +0000 UTC m=+19.292447311" Nov 02 01:15:19.405151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.405096 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz"] Nov 02 01:15:19.408567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.408543 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.416072 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.415931 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 01:15:19.417591 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.417560 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-msdtj"] Nov 02 01:15:19.422621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.422596 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.427698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.427675 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz"] Nov 02 01:15:19.462141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462097 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-root\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462241 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.462329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462283 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzv5\" (UniqueName: \"kubernetes.io/projected/86828f31-5a23-4d6f-a771-fbf67b94f89b-kube-api-access-8lzv5\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462324 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/305e6b99-4656-459a-95a4-c98da237618f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.462494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462373 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/305e6b99-4656-459a-95a4-c98da237618f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.462494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462405 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-accelerators-collector-config\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462433 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-wtmp\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462466 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-sys\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462493 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86828f31-5a23-4d6f-a771-fbf67b94f89b-metrics-client-ca\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462519 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462548 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nr2\" (UniqueName: \"kubernetes.io/projected/305e6b99-4656-459a-95a4-c98da237618f-kube-api-access-q2nr2\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.462726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462572 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-textfile\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.462726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.462610 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-tls\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.470309 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.470157 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.470309 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.470221 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.472480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.472317 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 01:15:19.472480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.472331 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 01:15:19.472480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.472368 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 01:15:19.472480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.472376 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 01:15:19.472480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.472427 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6ktbv\"" Nov 02 01:15:19.472480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.472461 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pc7pp\"" Nov 02 01:15:19.488801 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.488672 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 01:15:19.488962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.488818 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 01:15:19.488962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.488924 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 01:15:19.488962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.488946 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 01:15:19.571050 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571009 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/305e6b99-4656-459a-95a4-c98da237618f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571060 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-accelerators-collector-config\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571090 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-wtmp\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571135 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-sys\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571163 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86828f31-5a23-4d6f-a771-fbf67b94f89b-metrics-client-ca\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571190 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571220 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nr2\" (UniqueName: \"kubernetes.io/projected/305e6b99-4656-459a-95a4-c98da237618f-kube-api-access-q2nr2\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571242 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-textfile\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571263 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-tls\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571282 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571303 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571323 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-root\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571341 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571361 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzv5\" (UniqueName: \"kubernetes.io/projected/86828f31-5a23-4d6f-a771-fbf67b94f89b-kube-api-access-8lzv5\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.571584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571382 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/305e6b99-4656-459a-95a4-c98da237618f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571944 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571673 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/305e6b99-4656-459a-95a4-c98da237618f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.571944 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571726 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/305e6b99-4656-459a-95a4-c98da237618f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.572029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.571956 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-textfile\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.572029 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:19.571968 4552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Nov 02 01:15:19.572149 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:19.572033 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-tls podName:86828f31-5a23-4d6f-a771-fbf67b94f89b nodeName:}" failed. No retries permitted until 2026-11-02 01:15:20.072013566 +0000 UTC m=+20.177802473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-tls") pod "node-exporter-msdtj" (UID: "86828f31-5a23-4d6f-a771-fbf67b94f89b") : secret "node-exporter-tls" not found Nov 02 01:15:19.572149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.572110 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-accelerators-collector-config\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.572259 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.572204 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-root\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.572402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.572381 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86828f31-5a23-4d6f-a771-fbf67b94f89b-metrics-client-ca\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.572470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.572439 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-wtmp\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.572517 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.572495 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86828f31-5a23-4d6f-a771-fbf67b94f89b-sys\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.572930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.572908 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.579505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.574884 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.579505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.575380 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.579505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.575380 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/305e6b99-4656-459a-95a4-c98da237618f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.584953 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.584931 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzv5\" (UniqueName: \"kubernetes.io/projected/86828f31-5a23-4d6f-a771-fbf67b94f89b-kube-api-access-8lzv5\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:19.585274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.585252 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nr2\" (UniqueName: \"kubernetes.io/projected/305e6b99-4656-459a-95a4-c98da237618f-kube-api-access-q2nr2\") pod \"kube-state-metrics-5dc8d7d89b-lq2pz\" (UID: \"305e6b99-4656-459a-95a4-c98da237618f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:19.739075 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:19.739041 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" Nov 02 01:15:20.024908 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.024805 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz"] Nov 02 01:15:20.029152 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:20.029097 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305e6b99_4656_459a_95a4_c98da237618f.slice/crio-d04b88be211dde62a1d8f6f1d7672570f8dd31a7ee4013ce5a06960a5c3a37a1 WatchSource:0}: Error finding container d04b88be211dde62a1d8f6f1d7672570f8dd31a7ee4013ce5a06960a5c3a37a1: Status 404 returned error can't find the container with id d04b88be211dde62a1d8f6f1d7672570f8dd31a7ee4013ce5a06960a5c3a37a1 Nov 02 01:15:20.075006 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.074971 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-tls\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:20.077466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.077444 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86828f31-5a23-4d6f-a771-fbf67b94f89b-node-exporter-tls\") pod \"node-exporter-msdtj\" (UID: \"86828f31-5a23-4d6f-a771-fbf67b94f89b\") " pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:20.168736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.168695 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" event={"ID":"305e6b99-4656-459a-95a4-c98da237618f","Type":"ContainerStarted","Data":"d04b88be211dde62a1d8f6f1d7672570f8dd31a7ee4013ce5a06960a5c3a37a1"} Nov 02 01:15:20.351408 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.351334 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-msdtj" Nov 02 01:15:20.366258 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:20.366223 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86828f31_5a23_4d6f_a771_fbf67b94f89b.slice/crio-c9e55fd91bfbfff11d4e51b7b0f8980f7c89245ce846fb033d8a11e9d1b99378 WatchSource:0}: Error finding container c9e55fd91bfbfff11d4e51b7b0f8980f7c89245ce846fb033d8a11e9d1b99378: Status 404 returned error can't find the container with id c9e55fd91bfbfff11d4e51b7b0f8980f7c89245ce846fb033d8a11e9d1b99378 Nov 02 01:15:20.472959 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.472924 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:15:20.477455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.477437 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.481685 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.481634 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 01:15:20.481821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.481689 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 01:15:20.481821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.481789 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 01:15:20.481821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.481789 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 01:15:20.482642 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.482617 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 01:15:20.482725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.482657 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bstbs\"" Nov 02 01:15:20.482725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.482664 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 01:15:20.482725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.482663 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 01:15:20.482725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.482667 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 01:15:20.482725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.482692 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 01:15:20.497371 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.496956 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:15:20.578535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578496 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578555 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578662 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578701 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578743 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578792 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578818 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfgw\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578847 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.578891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578873 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.579138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578897 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.579138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578932 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.579138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578959 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.579138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.578988 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680388 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfgw\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680430 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680452 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680588 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680631 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680662 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680689 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680739 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680775 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680803 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680828 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680872 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.680935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.680923 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.682252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.682223 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.682431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.682295 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.682431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.682352 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.683978 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.683952 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.684490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.684466 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.685228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.685188 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.685327 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.685237 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.685393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.685358 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.685559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.685526 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.685970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.685944 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.686093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.686073 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.686505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.686482 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.692833 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.692811 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfgw\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw\") pod \"alertmanager-main-0\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:20.805950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:20.805908 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:15:21.118414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:21.118355 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:15:21.176931 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:21.176884 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"80b5b8a25b113ef0ad28ed8ee4b80b1b90730c4cf82e59f19fa5c78cb20af10c"} Nov 02 01:15:21.179308 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:21.179273 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-msdtj" event={"ID":"86828f31-5a23-4d6f-a771-fbf67b94f89b","Type":"ContainerStarted","Data":"c9e55fd91bfbfff11d4e51b7b0f8980f7c89245ce846fb033d8a11e9d1b99378"} Nov 02 01:15:22.187510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:22.186349 4552 generic.go:356] "Generic (PLEG): container finished" podID="86828f31-5a23-4d6f-a771-fbf67b94f89b" containerID="0694d8dc04e783dbfae7fd6e5d3681f59a84771286d1025af20e553253696cfc" exitCode=0 Nov 02 01:15:22.187510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:22.186512 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-msdtj" event={"ID":"86828f31-5a23-4d6f-a771-fbf67b94f89b","Type":"ContainerDied","Data":"0694d8dc04e783dbfae7fd6e5d3681f59a84771286d1025af20e553253696cfc"} Nov 02 01:15:22.190438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:22.190402 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" event={"ID":"305e6b99-4656-459a-95a4-c98da237618f","Type":"ContainerStarted","Data":"9b5e7efe34f6ddef00c917f44db02a4518822406f9db4f43855f68b8c97f8783"} Nov 02 01:15:22.190559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:22.190476 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" event={"ID":"305e6b99-4656-459a-95a4-c98da237618f","Type":"ContainerStarted","Data":"bbe3280ce69533fb66ef462aecc85cae243e501ecdb366cde880384db6d4c410"} Nov 02 01:15:22.190559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:22.190505 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" event={"ID":"305e6b99-4656-459a-95a4-c98da237618f","Type":"ContainerStarted","Data":"01ea6e8c0dac3f148d9d7b2c24d4ac7ca814e201094c87c2d083fcbb192cfcdb"} Nov 02 01:15:22.457301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:22.457234 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qjmzq" Nov 02 01:15:23.195985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.195952 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="60757fd64b6af6eec1e0a98d1b65ba70b10dc7e993f30f7199e622195efef9a3" exitCode=0 Nov 02 01:15:23.196483 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.196045 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"60757fd64b6af6eec1e0a98d1b65ba70b10dc7e993f30f7199e622195efef9a3"} Nov 02 01:15:23.200263 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.200242 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-msdtj" event={"ID":"86828f31-5a23-4d6f-a771-fbf67b94f89b","Type":"ContainerStarted","Data":"24ec136aa8d9d3f8dd42953d3f9819220e11a07c88d281190f38ec988bc374a2"} Nov 02 01:15:23.200366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.200286 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-msdtj" event={"ID":"86828f31-5a23-4d6f-a771-fbf67b94f89b","Type":"ContainerStarted","Data":"b2411d5c2c6ece687a6a7f732521defcbe99e5f2472d48a6072c2af24ab0b113"} Nov 02 01:15:23.230864 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.230501 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lq2pz" podStartSLOduration=2.528101679 podStartE2EDuration="4.230482161s" totalImagesPullingTime="1.702380482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:22.23235108 +0000 UTC m=+22.338139994" watchObservedRunningTime="2026-11-02 01:15:23.230482161 +0000 UTC m=+23.336271077" Nov 02 01:15:23.252986 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.252939 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-msdtj" podStartSLOduration=2.975219467 podStartE2EDuration="4.252925929s" totalImagesPullingTime="1.186990946s" totalInitContainerRuntime="90.715516ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:23.251035148 +0000 UTC m=+23.356824076" watchObservedRunningTime="2026-11-02 01:15:23.252925929 +0000 UTC m=+23.358714842" Nov 02 01:15:23.782136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.782088 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c4dfbbfb-bj6wn"] Nov 02 01:15:23.785295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.785278 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.788840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.788819 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 01:15:23.788840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.788833 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 01:15:23.796339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.796111 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c4dfbbfb-bj6wn"] Nov 02 01:15:23.818210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.818183 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 01:15:23.818354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.818322 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 01:15:23.818493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.818466 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cdm85kucj7s11\"" Nov 02 01:15:23.818493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.818445 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dl7lv\"" Nov 02 01:15:23.915902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.915867 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98370bbf-d509-4f16-9ca2-d7f16526eee1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.916059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.915916 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98370bbf-d509-4f16-9ca2-d7f16526eee1-metrics-server-audit-profiles\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.916059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.915954 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-client-ca-bundle\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.916059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.915986 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-secret-metrics-server-client-certs\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.916059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.916022 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rnmw\" (UniqueName: \"kubernetes.io/projected/98370bbf-d509-4f16-9ca2-d7f16526eee1-kube-api-access-5rnmw\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.916262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.916076 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-secret-metrics-server-tls\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:23.916262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:23.916133 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98370bbf-d509-4f16-9ca2-d7f16526eee1-audit-log\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016607 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-secret-metrics-server-client-certs\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016647 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rnmw\" (UniqueName: \"kubernetes.io/projected/98370bbf-d509-4f16-9ca2-d7f16526eee1-kube-api-access-5rnmw\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016697 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-secret-metrics-server-tls\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016717 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98370bbf-d509-4f16-9ca2-d7f16526eee1-audit-log\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016746 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98370bbf-d509-4f16-9ca2-d7f16526eee1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016772 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98370bbf-d509-4f16-9ca2-d7f16526eee1-metrics-server-audit-profiles\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.016809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.016801 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-client-ca-bundle\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.017315 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.017268 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98370bbf-d509-4f16-9ca2-d7f16526eee1-audit-log\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.017745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.017720 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98370bbf-d509-4f16-9ca2-d7f16526eee1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.017921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.017902 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98370bbf-d509-4f16-9ca2-d7f16526eee1-metrics-server-audit-profiles\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.019691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.019673 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-client-ca-bundle\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.019887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.019869 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-secret-metrics-server-client-certs\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.019953 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.019927 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98370bbf-d509-4f16-9ca2-d7f16526eee1-secret-metrics-server-tls\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.029878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.029852 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rnmw\" (UniqueName: \"kubernetes.io/projected/98370bbf-d509-4f16-9ca2-d7f16526eee1-kube-api-access-5rnmw\") pod \"metrics-server-c4dfbbfb-bj6wn\" (UID: \"98370bbf-d509-4f16-9ca2-d7f16526eee1\") " pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.097082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.097002 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:24.164531 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.164259 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx"] Nov 02 01:15:24.176748 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.173045 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.178018 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.177980 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 01:15:24.181099 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.181071 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx"] Nov 02 01:15:24.187075 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.187032 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6jgzh\"" Nov 02 01:15:24.319538 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.319493 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf628013-f478-4707-ab31-c9cd6e5ed0b6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4gjx\" (UID: \"bf628013-f478-4707-ab31-c9cd6e5ed0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.320028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.319606 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhzl\" (UniqueName: \"kubernetes.io/projected/bf628013-f478-4707-ab31-c9cd6e5ed0b6-kube-api-access-9rhzl\") pod \"monitoring-plugin-77f4f76594-p4gjx\" (UID: \"bf628013-f478-4707-ab31-c9cd6e5ed0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.399059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.398971 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c4dfbbfb-bj6wn"] Nov 02 01:15:24.402870 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:24.402839 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98370bbf_d509_4f16_9ca2_d7f16526eee1.slice/crio-a2036c607c9c80b8130d45822c119e3409ea7d4b8b41354c952bc200258ff4c5 WatchSource:0}: Error finding container a2036c607c9c80b8130d45822c119e3409ea7d4b8b41354c952bc200258ff4c5: Status 404 returned error can't find the container with id a2036c607c9c80b8130d45822c119e3409ea7d4b8b41354c952bc200258ff4c5 Nov 02 01:15:24.420369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.420343 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhzl\" (UniqueName: \"kubernetes.io/projected/bf628013-f478-4707-ab31-c9cd6e5ed0b6-kube-api-access-9rhzl\") pod \"monitoring-plugin-77f4f76594-p4gjx\" (UID: \"bf628013-f478-4707-ab31-c9cd6e5ed0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.420468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.420446 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf628013-f478-4707-ab31-c9cd6e5ed0b6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4gjx\" (UID: \"bf628013-f478-4707-ab31-c9cd6e5ed0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.423247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.423217 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf628013-f478-4707-ab31-c9cd6e5ed0b6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4gjx\" (UID: \"bf628013-f478-4707-ab31-c9cd6e5ed0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.435277 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.435252 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhzl\" (UniqueName: \"kubernetes.io/projected/bf628013-f478-4707-ab31-c9cd6e5ed0b6-kube-api-access-9rhzl\") pod \"monitoring-plugin-77f4f76594-p4gjx\" (UID: \"bf628013-f478-4707-ab31-c9cd6e5ed0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.480559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.480525 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:24.480711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.480578 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:24.486292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.486267 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:24.497217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.497190 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:24.587361 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.587321 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx"] Nov 02 01:15:24.592395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.592368 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.596289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.596260 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 01:15:24.596289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.596272 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 01:15:24.596289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.596278 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 01:15:24.596524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.596316 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 01:15:24.596524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.596343 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 01:15:24.596524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.596434 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4rj5l\"" Nov 02 01:15:24.600423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.600400 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 01:15:24.611083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.611059 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx"] Nov 02 01:15:24.722061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722018 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-metrics-client-ca\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722078 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-serving-certs-ca-bundle\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722132 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zktd\" (UniqueName: \"kubernetes.io/projected/2724697a-5273-4f4a-995c-0515795bea22-kube-api-access-9zktd\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722169 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-telemeter-client-tls\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722198 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-federate-client-tls\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722221 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722244 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.722566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.722288 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-secret-telemeter-client\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.777629 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.777604 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx"] Nov 02 01:15:24.823454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823419 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-metrics-client-ca\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823471 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-serving-certs-ca-bundle\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823514 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zktd\" (UniqueName: \"kubernetes.io/projected/2724697a-5273-4f4a-995c-0515795bea22-kube-api-access-9zktd\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823542 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-telemeter-client-tls\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823575 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-federate-client-tls\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823601 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823625 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.823961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.823671 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-secret-telemeter-client\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.824286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.824259 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-metrics-client-ca\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.824377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.824308 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-serving-certs-ca-bundle\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.825573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.825545 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2724697a-5273-4f4a-995c-0515795bea22-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.827110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.826876 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.827110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.827070 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-secret-telemeter-client\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.827110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.827098 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-federate-client-tls\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.827794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.827771 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2724697a-5273-4f4a-995c-0515795bea22-telemeter-client-tls\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.836774 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.836753 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zktd\" (UniqueName: \"kubernetes.io/projected/2724697a-5273-4f4a-995c-0515795bea22-kube-api-access-9zktd\") pod \"telemeter-client-5b55d9fb6d-qg8xx\" (UID: \"2724697a-5273-4f4a-995c-0515795bea22\") " pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:24.909418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:24.909379 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" Nov 02 01:15:25.146679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.146638 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86949d8f76-n45rv"] Nov 02 01:15:25.150908 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.150883 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.160970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.160947 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86949d8f76-n45rv"] Nov 02 01:15:25.161235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.161218 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 01:15:25.217742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.217700 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" event={"ID":"98370bbf-d509-4f16-9ca2-d7f16526eee1","Type":"ContainerStarted","Data":"a2036c607c9c80b8130d45822c119e3409ea7d4b8b41354c952bc200258ff4c5"} Nov 02 01:15:25.219689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.219650 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" event={"ID":"bf628013-f478-4707-ab31-c9cd6e5ed0b6","Type":"ContainerStarted","Data":"b6f570019f82a8cf81861bf243b3c1601a87b8b6ef5c910cf886afdbbd4ec0f6"} Nov 02 01:15:25.225075 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.225028 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:15:25.226755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.226727 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx"] Nov 02 01:15:25.227056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227040 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227069 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227095 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227174 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227208 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227292 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227358 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.227742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.227395 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbgs\" (UniqueName: \"kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328225 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328286 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328345 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328397 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328422 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328450 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328544 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.329829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.328581 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbgs\" (UniqueName: \"kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.331231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.330701 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.331231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.330882 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.331551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.331526 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.331980 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.331881 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.332065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.332012 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.334300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.334257 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.334623 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.334598 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.349008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.348982 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbgs\" (UniqueName: \"kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs\") pod \"console-86949d8f76-n45rv\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.468576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.468410 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:25.768637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:25.768597 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86949d8f76-n45rv"] Nov 02 01:15:25.772814 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:25.772782 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e7dbe3_8aae_438c_8369_351e74872175.slice/crio-ace84671f61fe1d5915ef7f0d14496731e8a4d1966190dd41f61dda3b2bd9dba WatchSource:0}: Error finding container ace84671f61fe1d5915ef7f0d14496731e8a4d1966190dd41f61dda3b2bd9dba: Status 404 returned error can't find the container with id ace84671f61fe1d5915ef7f0d14496731e8a4d1966190dd41f61dda3b2bd9dba Nov 02 01:15:26.229384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.229349 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86949d8f76-n45rv" event={"ID":"31e7dbe3-8aae-438c-8369-351e74872175","Type":"ContainerStarted","Data":"56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e"} Nov 02 01:15:26.229546 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.229409 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86949d8f76-n45rv" event={"ID":"31e7dbe3-8aae-438c-8369-351e74872175","Type":"ContainerStarted","Data":"ace84671f61fe1d5915ef7f0d14496731e8a4d1966190dd41f61dda3b2bd9dba"} Nov 02 01:15:26.235723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.235671 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" event={"ID":"2724697a-5273-4f4a-995c-0515795bea22","Type":"ContainerStarted","Data":"018dae71e786e330231c5fb87e83e1a19872eaccf5664a09710e3625fd212ca0"} Nov 02 01:15:26.242913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.242872 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"cd1e428b3ef15d7acd2615a282fa3c4506f037ce52a28e6aae1b4aeee0bedc33"} Nov 02 01:15:26.243046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.242930 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"9a55d48b43c87a7daf2f2c3383e8d690c10ab7b3d95972c1fe1813f659042200"} Nov 02 01:15:26.243046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.242964 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"7c157d27ee5d3da19839eb0ebeaba457fd67952cd52d0901f267be61bcaba6c6"} Nov 02 01:15:26.243046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.243004 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"03795fe919391a7f9baa778d723ce795b637ba0c5a7b224ae0c948fc0e3a22c1"} Nov 02 01:15:26.243046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:26.243033 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"65d0cd248f9c2fca2c250c6a8eddea953d825e276d0ff1860b61c675e03b592b"} Nov 02 01:15:27.249273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.249183 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" event={"ID":"98370bbf-d509-4f16-9ca2-d7f16526eee1","Type":"ContainerStarted","Data":"278bed836a77720d18a22827ad5e315bd3ba6efa0470b3c7ef1b1e395257bbe6"} Nov 02 01:15:27.251716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.251683 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" event={"ID":"bf628013-f478-4707-ab31-c9cd6e5ed0b6","Type":"ContainerStarted","Data":"19470c8ba3b1fbfca1c422b1aa5e9c9610befd4c9e2638dd30d1dd1c271be89c"} Nov 02 01:15:27.251915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.251897 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:27.256606 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.256522 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerStarted","Data":"4ece11b6ceb6267e5d5b9e3d09c6130468743672fea93ef705e4c892ed91299b"} Nov 02 01:15:27.258564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.258542 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" Nov 02 01:15:27.270030 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.269974 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86949d8f76-n45rv" podStartSLOduration=2.269957315 podStartE2EDuration="2.269957315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:26.251320986 +0000 UTC m=+26.357109901" watchObservedRunningTime="2026-11-02 01:15:27.269957315 +0000 UTC m=+27.375746231" Nov 02 01:15:27.270809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.270765 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" podStartSLOduration=1.962954289 podStartE2EDuration="4.270755203s" totalImagesPullingTime="2.307800914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:27.269486622 +0000 UTC m=+27.375275578" watchObservedRunningTime="2026-11-02 01:15:27.270755203 +0000 UTC m=+27.376544118" Nov 02 01:15:27.291808 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.291739 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4gjx" podStartSLOduration=1.2019934939999999 podStartE2EDuration="3.29172312s" totalImagesPullingTime="2.089729626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:27.291285779 +0000 UTC m=+27.397074692" watchObservedRunningTime="2026-11-02 01:15:27.29172312 +0000 UTC m=+27.397512037" Nov 02 01:15:27.329680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:27.329618 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.072216393 podStartE2EDuration="7.329598591s" totalImagesPullingTime="5.240711744s" totalInitContainerRuntime="16.670454ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:27.326647598 +0000 UTC m=+27.432436511" watchObservedRunningTime="2026-11-02 01:15:27.329598591 +0000 UTC m=+27.435387506" Nov 02 01:15:27.758348 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:27.758318 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:15:28.262755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:28.262716 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" event={"ID":"2724697a-5273-4f4a-995c-0515795bea22","Type":"ContainerStarted","Data":"c04e13da779c1416b9c7733bf17713137d80644243e590a9f6ba6ab0b3799e21"} Nov 02 01:15:28.263200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:28.262787 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" event={"ID":"2724697a-5273-4f4a-995c-0515795bea22","Type":"ContainerStarted","Data":"f3d8e02f62b136fd79f4e93d217987afbe083a315ef54710b07182c07a4761da"} Nov 02 01:15:28.263200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:28.262815 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" event={"ID":"2724697a-5273-4f4a-995c-0515795bea22","Type":"ContainerStarted","Data":"df14f2012f1bcf351b5dc53abc756e3facaa51cd03099c6da3b814c9a0f18b5e"} Nov 02 01:15:28.291485 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:28.291417 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b55d9fb6d-qg8xx" podStartSLOduration=1.932508263 podStartE2EDuration="4.291398504s" totalImagesPullingTime="2.358890241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:28.289542734 +0000 UTC m=+28.395331649" watchObservedRunningTime="2026-11-02 01:15:28.291398504 +0000 UTC m=+28.397187419" Nov 02 01:15:32.084789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:32.084756 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4x747" Nov 02 01:15:34.293770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.290548 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qj4wc" event={"ID":"ec0430a9-64d9-40ed-94b3-5ad4c9c826b8","Type":"ContainerStarted","Data":"d156a05a1487f71adb62dde62c7b4750968209b6fba9df9a24e0ab0fcf8c5839"} Nov 02 01:15:34.294439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.293815 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:34.313053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.312987 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-qj4wc" podStartSLOduration=1.706215556 podStartE2EDuration="22.312969019s" totalImagesPullingTime="20.606753463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:34.311145011 +0000 UTC m=+34.416933926" watchObservedRunningTime="2026-11-02 01:15:34.312969019 +0000 UTC m=+34.418757935" Nov 02 01:15:34.313689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.313660 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-qj4wc" Nov 02 01:15:34.556568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.556487 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86949d8f76-n45rv"] Nov 02 01:15:34.586318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.586278 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7585fddb94-8ktbt"] Nov 02 01:15:34.608244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.608209 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7585fddb94-8ktbt"] Nov 02 01:15:34.608425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.608350 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726189 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726241 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726374 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726538 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726470 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrw4\" (UniqueName: \"kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726559 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726638 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726605 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726638 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726634 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.726721 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.726684 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827675 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827579 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827675 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827650 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827687 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827743 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827773 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827807 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827850 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.827912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.827887 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrw4\" (UniqueName: \"kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.828512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.828411 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.828512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.828474 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.828917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.828896 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.828990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.828934 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.829218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.829196 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.831097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.831074 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.831832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.831807 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.840912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.840880 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrw4\" (UniqueName: \"kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4\") pod \"console-7585fddb94-8ktbt\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:34.935088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:34.935041 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:35.246242 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:35.246208 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7585fddb94-8ktbt"] Nov 02 01:15:35.250045 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:15:35.250003 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09af22f1_5a55_4e91_8ec4_32bd81ac002c.slice/crio-2c986fa6e0f9b4f3193045f738f451e1b3f68d1d5b127251d47c9d162ae628ab WatchSource:0}: Error finding container 2c986fa6e0f9b4f3193045f738f451e1b3f68d1d5b127251d47c9d162ae628ab: Status 404 returned error can't find the container with id 2c986fa6e0f9b4f3193045f738f451e1b3f68d1d5b127251d47c9d162ae628ab Nov 02 01:15:35.297835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:35.297770 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585fddb94-8ktbt" event={"ID":"09af22f1-5a55-4e91-8ec4-32bd81ac002c","Type":"ContainerStarted","Data":"2c986fa6e0f9b4f3193045f738f451e1b3f68d1d5b127251d47c9d162ae628ab"} Nov 02 01:15:35.469416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:35.469370 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:36.304929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:36.304884 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585fddb94-8ktbt" event={"ID":"09af22f1-5a55-4e91-8ec4-32bd81ac002c","Type":"ContainerStarted","Data":"ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a"} Nov 02 01:15:36.327856 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:36.327787 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7585fddb94-8ktbt" podStartSLOduration=2.327764409 podStartE2EDuration="2.327764409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:36.326981041 +0000 UTC m=+36.432769957" watchObservedRunningTime="2026-11-02 01:15:36.327764409 +0000 UTC m=+36.433553324" Nov 02 01:15:44.097599 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:44.097517 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:44.097599 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:44.097575 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:15:44.935372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:44.935335 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:44.935547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:44.935389 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:44.940179 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:44.940156 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:45.356562 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:45.356536 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:15:45.407352 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:45.407316 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c6d5d46b6-grhbf"] Nov 02 01:15:57.749415 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:15:57.749377 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:15:59.581407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.581353 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86949d8f76-n45rv" podUID="31e7dbe3-8aae-438c-8369-351e74872175" containerName="console" containerID="cri-o://56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e" gracePeriod=15 Nov 02 01:15:59.863023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.862945 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:15:59.955921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.955807 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.955991 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs\" (UniqueName: \"kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956040 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956077 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp\" (UniqueName: \"kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956094 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956155 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956186 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956192 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:15:59.956365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956213 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config\") pod \"31e7dbe3-8aae-438c-8369-351e74872175\" (UID: \"31e7dbe3-8aae-438c-8369-351e74872175\") " Nov 02 01:15:59.956560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956445 4552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:15:59.956560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956492 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:15:59.956662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956526 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:15:59.956716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956664 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:15:59.956765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.956713 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:15:59.958575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.958549 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:15:59.958575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.958556 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "kube-api-access-9qbgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:15:59.958712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:15:59.958571 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert" pod "31e7dbe3-8aae-438c-8369-351e74872175" (UID: "31e7dbe3-8aae-438c-8369-351e74872175"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:00.057386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057352 4552 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31e7dbe3-8aae-438c-8369-351e74872175-tmp\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.057386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057378 4552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-trusted-ca-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.057386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057388 4552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-serving-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.057595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057398 4552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-console-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.057595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057408 4552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31e7dbe3-8aae-438c-8369-351e74872175-console-oauth-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.057595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057416 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qbgs\" (UniqueName: \"kubernetes.io/projected/31e7dbe3-8aae-438c-8369-351e74872175-kube-api-access-9qbgs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.057595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.057425 4552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31e7dbe3-8aae-438c-8369-351e74872175-oauth-serving-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:00.411970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.411933 4552 generic.go:356] "Generic (PLEG): container finished" podID="31e7dbe3-8aae-438c-8369-351e74872175" containerID="56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e" exitCode=0 Nov 02 01:16:00.412160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.411977 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86949d8f76-n45rv" Nov 02 01:16:00.412160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.411999 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86949d8f76-n45rv" event={"ID":"31e7dbe3-8aae-438c-8369-351e74872175","Type":"ContainerDied","Data":"56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e"} Nov 02 01:16:00.412160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.412045 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86949d8f76-n45rv" event={"ID":"31e7dbe3-8aae-438c-8369-351e74872175","Type":"ContainerDied","Data":"ace84671f61fe1d5915ef7f0d14496731e8a4d1966190dd41f61dda3b2bd9dba"} Nov 02 01:16:00.412160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.412076 4552 scope.go:122] "RemoveContainer" containerID="56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e" Nov 02 01:16:00.420400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.420376 4552 scope.go:122] "RemoveContainer" containerID="56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e" Nov 02 01:16:00.420675 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:16:00.420655 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e\": container with ID starting with 56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e not found: ID does not exist" containerID="56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e" Nov 02 01:16:00.420718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.420688 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e"} err="failed to get container status \"56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e\": rpc error: code = NotFound desc = could not find container \"56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e\": container with ID starting with 56dc77894bd935790a00957d9d7488b67df899fee693caabec2d0f43641f481e not found: ID does not exist" Nov 02 01:16:00.429924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.429904 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86949d8f76-n45rv"] Nov 02 01:16:00.433588 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:00.433565 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86949d8f76-n45rv"] Nov 02 01:16:02.041587 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:02.041554 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e7dbe3-8aae-438c-8369-351e74872175" path="/var/lib/kubelet/pods/31e7dbe3-8aae-438c-8369-351e74872175/volumes" Nov 02 01:16:04.103001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:04.102966 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:16:04.106664 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:04.106643 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c4dfbbfb-bj6wn" Nov 02 01:16:08.424791 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.424752 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c8fb7895f-dxl9d"] Nov 02 01:16:08.425274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.425085 4552 cpu_manager.go:414] "containerMap: removing container" podUID="31e7dbe3-8aae-438c-8369-351e74872175" containerName="console" Nov 02 01:16:08.425274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.425103 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e7dbe3-8aae-438c-8369-351e74872175" containerName="console" Nov 02 01:16:08.425274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.425191 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e7dbe3-8aae-438c-8369-351e74872175" containerName="console" Nov 02 01:16:08.443187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.443160 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8fb7895f-dxl9d"] Nov 02 01:16:08.443324 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.443256 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534256 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-serving-cert\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534290 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-config\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534499 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534315 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqq5j\" (UniqueName: \"kubernetes.io/projected/8c5c5331-9226-40e4-80fa-d5a7053376fb-kube-api-access-wqq5j\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534499 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534451 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-service-ca\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534499 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534478 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c5c5331-9226-40e4-80fa-d5a7053376fb-tmp\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534549 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-oauth-config\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534588 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-oauth-serving-cert\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.534657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.534620 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-trusted-ca-bundle\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635591 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-oauth-serving-cert\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635632 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-trusted-ca-bundle\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635656 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-serving-cert\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635672 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-config\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635706 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqq5j\" (UniqueName: \"kubernetes.io/projected/8c5c5331-9226-40e4-80fa-d5a7053376fb-kube-api-access-wqq5j\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635750 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-service-ca\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635771 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c5c5331-9226-40e4-80fa-d5a7053376fb-tmp\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.635994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.635819 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-oauth-config\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.636232 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.636208 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c5c5331-9226-40e4-80fa-d5a7053376fb-tmp\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.636531 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.636502 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-oauth-serving-cert\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.636778 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.636543 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-trusted-ca-bundle\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.636778 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.636565 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-service-ca\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.636778 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.636708 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-config\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.638415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.638390 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-oauth-config\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.638599 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.638581 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c5c5331-9226-40e4-80fa-d5a7053376fb-console-serving-cert\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.648377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.648358 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqq5j\" (UniqueName: \"kubernetes.io/projected/8c5c5331-9226-40e4-80fa-d5a7053376fb-kube-api-access-wqq5j\") pod \"console-6c8fb7895f-dxl9d\" (UID: \"8c5c5331-9226-40e4-80fa-d5a7053376fb\") " pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:08.765902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:08.765858 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:09.069060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.069037 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8fb7895f-dxl9d"] Nov 02 01:16:09.071594 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:16:09.071565 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5c5331_9226_40e4_80fa_d5a7053376fb.slice/crio-851c50e2583d45c7f103e153f87e349aaf77c0c0d8f5dd99683eefc853ef92af WatchSource:0}: Error finding container 851c50e2583d45c7f103e153f87e349aaf77c0c0d8f5dd99683eefc853ef92af: Status 404 returned error can't find the container with id 851c50e2583d45c7f103e153f87e349aaf77c0c0d8f5dd99683eefc853ef92af Nov 02 01:16:09.190053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190013 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:16:09.190332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190309 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="alertmanager" containerID="cri-o://65d0cd248f9c2fca2c250c6a8eddea953d825e276d0ff1860b61c675e03b592b" gracePeriod=120 Nov 02 01:16:09.190409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190385 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-web" containerID="cri-o://7c157d27ee5d3da19839eb0ebeaba457fd67952cd52d0901f267be61bcaba6c6" gracePeriod=120 Nov 02 01:16:09.190465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190418 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="prom-label-proxy" containerID="cri-o://4ece11b6ceb6267e5d5b9e3d09c6130468743672fea93ef705e4c892ed91299b" gracePeriod=120 Nov 02 01:16:09.190524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190395 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy" containerID="cri-o://9a55d48b43c87a7daf2f2c3383e8d690c10ab7b3d95972c1fe1813f659042200" gracePeriod=120 Nov 02 01:16:09.190524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190388 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-metric" containerID="cri-o://cd1e428b3ef15d7acd2615a282fa3c4506f037ce52a28e6aae1b4aeee0bedc33" gracePeriod=120 Nov 02 01:16:09.190644 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.190419 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="config-reloader" containerID="cri-o://03795fe919391a7f9baa778d723ce795b637ba0c5a7b224ae0c948fc0e3a22c1" gracePeriod=120 Nov 02 01:16:09.450233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.450195 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8fb7895f-dxl9d" event={"ID":"8c5c5331-9226-40e4-80fa-d5a7053376fb","Type":"ContainerStarted","Data":"b263de5d6ab0f05b6e54b66390ddb724b309b5035d8bcebba3020b5e01a83cf4"} Nov 02 01:16:09.450716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.450253 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8fb7895f-dxl9d" event={"ID":"8c5c5331-9226-40e4-80fa-d5a7053376fb","Type":"ContainerStarted","Data":"851c50e2583d45c7f103e153f87e349aaf77c0c0d8f5dd99683eefc853ef92af"} Nov 02 01:16:09.453655 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453629 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="4ece11b6ceb6267e5d5b9e3d09c6130468743672fea93ef705e4c892ed91299b" exitCode=0 Nov 02 01:16:09.453765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453656 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="cd1e428b3ef15d7acd2615a282fa3c4506f037ce52a28e6aae1b4aeee0bedc33" exitCode=0 Nov 02 01:16:09.453765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453669 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="9a55d48b43c87a7daf2f2c3383e8d690c10ab7b3d95972c1fe1813f659042200" exitCode=0 Nov 02 01:16:09.453765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453675 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="03795fe919391a7f9baa778d723ce795b637ba0c5a7b224ae0c948fc0e3a22c1" exitCode=0 Nov 02 01:16:09.453765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453681 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="65d0cd248f9c2fca2c250c6a8eddea953d825e276d0ff1860b61c675e03b592b" exitCode=0 Nov 02 01:16:09.453765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453710 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"4ece11b6ceb6267e5d5b9e3d09c6130468743672fea93ef705e4c892ed91299b"} Nov 02 01:16:09.453938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453803 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"cd1e428b3ef15d7acd2615a282fa3c4506f037ce52a28e6aae1b4aeee0bedc33"} Nov 02 01:16:09.453938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453837 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"9a55d48b43c87a7daf2f2c3383e8d690c10ab7b3d95972c1fe1813f659042200"} Nov 02 01:16:09.453938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453886 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"03795fe919391a7f9baa778d723ce795b637ba0c5a7b224ae0c948fc0e3a22c1"} Nov 02 01:16:09.453938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.453913 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"65d0cd248f9c2fca2c250c6a8eddea953d825e276d0ff1860b61c675e03b592b"} Nov 02 01:16:09.470223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:09.470170 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c8fb7895f-dxl9d" podStartSLOduration=1.470146197 podStartE2EDuration="1.470146197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:16:09.469881328 +0000 UTC m=+69.575670243" watchObservedRunningTime="2026-11-02 01:16:09.470146197 +0000 UTC m=+69.575935109" Nov 02 01:16:10.433410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.433350 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c6d5d46b6-grhbf" podUID="16e5a629-74d2-4e37-9867-cb9bd4f9bea9" containerName="console" containerID="cri-o://77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8" gracePeriod=15 Nov 02 01:16:10.461925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.461897 4552 generic.go:356] "Generic (PLEG): container finished" podID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerID="7c157d27ee5d3da19839eb0ebeaba457fd67952cd52d0901f267be61bcaba6c6" exitCode=0 Nov 02 01:16:10.462438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.461977 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"7c157d27ee5d3da19839eb0ebeaba457fd67952cd52d0901f267be61bcaba6c6"} Nov 02 01:16:10.718925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.718899 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:16:10.855866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.855791 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.855866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.855851 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.856077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.855881 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.856077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.855910 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.856077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.855993 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz\" (UniqueName: \"kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.856077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856069 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.856320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856104 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config\") pod \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\" (UID: \"16e5a629-74d2-4e37-9867-cb9bd4f9bea9\") " Nov 02 01:16:10.856320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856182 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:10.856320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856305 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:10.856505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856363 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:10.856505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856437 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:10.856505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856447 4552 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-tmp\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.856505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.856471 4552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.858416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.858391 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:10.858416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.858402 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "kube-api-access-x7lmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:10.858556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.858484 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert" pod "16e5a629-74d2-4e37-9867-cb9bd4f9bea9" (UID: "16e5a629-74d2-4e37-9867-cb9bd4f9bea9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:10.956929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.956899 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7lmz\" (UniqueName: \"kubernetes.io/projected/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-kube-api-access-x7lmz\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.956929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.956928 4552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-serving-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.957068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.956938 4552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-oauth-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.957068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.956947 4552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-oauth-serving-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.957068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.956955 4552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e5a629-74d2-4e37-9867-cb9bd4f9bea9-console-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:10.976413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:10.976384 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.057938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.057903 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.057945 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.057965 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.057997 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058079 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058239 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058298 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058332 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058361 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058396 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058425 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058423 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:11.058572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058452 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058489 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config\") pod \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\" (UID: \"c5c9abb4-26a9-469b-b161-2bde675fc7ba\") " Nov 02 01:16:11.058996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.058939 4552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-main-db\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.061005 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.060821 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.061005 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.060918 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:11.061005 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.060932 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:11.061343 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.061129 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:11.061471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.061226 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:11.061770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.061745 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.062626 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.062600 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.062971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.062941 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.063502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.063478 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "kube-api-access-vkfgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:11.064177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.064156 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.066347 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.066321 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.072663 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.072637 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config" pod "c5c9abb4-26a9-469b-b161-2bde675fc7ba" (UID: "c5c9abb4-26a9-469b-b161-2bde675fc7ba"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159869 4552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159898 4552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159908 4552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159918 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-metrics-client-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159928 4552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159936 4552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c9abb4-26a9-469b-b161-2bde675fc7ba-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159946 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkfgw\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-kube-api-access-vkfgw\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.159949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159955 4552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-cluster-tls-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.160305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159965 4552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5c9abb4-26a9-469b-b161-2bde675fc7ba-config-out\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.160305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159973 4552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-web-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.160305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159981 4552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5c9abb4-26a9-469b-b161-2bde675fc7ba-secret-alertmanager-main-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.160305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.159989 4552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5c9abb4-26a9-469b-b161-2bde675fc7ba-tls-assets\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:11.467021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.466986 4552 generic.go:356] "Generic (PLEG): container finished" podID="16e5a629-74d2-4e37-9867-cb9bd4f9bea9" containerID="77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8" exitCode=0 Nov 02 01:16:11.467439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.467039 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c6d5d46b6-grhbf" Nov 02 01:16:11.467439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.467062 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6d5d46b6-grhbf" event={"ID":"16e5a629-74d2-4e37-9867-cb9bd4f9bea9","Type":"ContainerDied","Data":"77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8"} Nov 02 01:16:11.467439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.467138 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c6d5d46b6-grhbf" event={"ID":"16e5a629-74d2-4e37-9867-cb9bd4f9bea9","Type":"ContainerDied","Data":"b1431c09b9b4265150c5abf4ac92e9c3a1065147cf86be58603765e4fc4b6a5f"} Nov 02 01:16:11.467439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.467162 4552 scope.go:122] "RemoveContainer" containerID="77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8" Nov 02 01:16:11.470592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.470560 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5c9abb4-26a9-469b-b161-2bde675fc7ba","Type":"ContainerDied","Data":"80b5b8a25b113ef0ad28ed8ee4b80b1b90730c4cf82e59f19fa5c78cb20af10c"} Nov 02 01:16:11.470727 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.470679 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.476474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.476459 4552 scope.go:122] "RemoveContainer" containerID="77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8" Nov 02 01:16:11.476745 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:16:11.476729 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8\": container with ID starting with 77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8 not found: ID does not exist" containerID="77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8" Nov 02 01:16:11.476799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.476754 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8"} err="failed to get container status \"77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8\": rpc error: code = NotFound desc = could not find container \"77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8\": container with ID starting with 77380ea843bd443856016e0cc6cee445cfd7ef98ed9962416e05968d253ba7a8 not found: ID does not exist" Nov 02 01:16:11.476799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.476771 4552 scope.go:122] "RemoveContainer" containerID="4ece11b6ceb6267e5d5b9e3d09c6130468743672fea93ef705e4c892ed91299b" Nov 02 01:16:11.488340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.488321 4552 scope.go:122] "RemoveContainer" containerID="cd1e428b3ef15d7acd2615a282fa3c4506f037ce52a28e6aae1b4aeee0bedc33" Nov 02 01:16:11.490794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.490775 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c6d5d46b6-grhbf"] Nov 02 01:16:11.494612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.494591 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c6d5d46b6-grhbf"] Nov 02 01:16:11.496327 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.496314 4552 scope.go:122] "RemoveContainer" containerID="9a55d48b43c87a7daf2f2c3383e8d690c10ab7b3d95972c1fe1813f659042200" Nov 02 01:16:11.503175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.503094 4552 scope.go:122] "RemoveContainer" containerID="7c157d27ee5d3da19839eb0ebeaba457fd67952cd52d0901f267be61bcaba6c6" Nov 02 01:16:11.507854 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.507820 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:16:11.511177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.510888 4552 scope.go:122] "RemoveContainer" containerID="03795fe919391a7f9baa778d723ce795b637ba0c5a7b224ae0c948fc0e3a22c1" Nov 02 01:16:11.512902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.512882 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:16:11.518423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.518405 4552 scope.go:122] "RemoveContainer" containerID="65d0cd248f9c2fca2c250c6a8eddea953d825e276d0ff1860b61c675e03b592b" Nov 02 01:16:11.525391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.525376 4552 scope.go:122] "RemoveContainer" containerID="60757fd64b6af6eec1e0a98d1b65ba70b10dc7e993f30f7199e622195efef9a3" Nov 02 01:16:11.543336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543313 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:16:11.543622 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543607 4552 cpu_manager.go:414] "containerMap: removing container" podUID="16e5a629-74d2-4e37-9867-cb9bd4f9bea9" containerName="console" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543623 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e5a629-74d2-4e37-9867-cb9bd4f9bea9" containerName="console" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543633 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="init-config-reloader" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543638 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="init-config-reloader" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543650 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-web" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543655 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-web" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543661 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-metric" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543666 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-metric" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543681 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="config-reloader" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543685 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="config-reloader" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543694 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy" Nov 02 01:16:11.543692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543699 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543705 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="alertmanager" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543709 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="alertmanager" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543720 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="prom-label-proxy" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543724 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="prom-label-proxy" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543766 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="prom-label-proxy" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543774 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e5a629-74d2-4e37-9867-cb9bd4f9bea9" containerName="console" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543781 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-web" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543786 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy-metric" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543794 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="config-reloader" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543799 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="kube-rbac-proxy" Nov 02 01:16:11.544213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.543804 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" containerName="alertmanager" Nov 02 01:16:11.573393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.573362 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:16:11.573565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.573516 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.577353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.577332 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 01:16:11.584543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.584523 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 01:16:11.664487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664455 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6e097c5-cc31-4829-888e-3837adf23172-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664492 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664517 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6e097c5-cc31-4829-888e-3837adf23172-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664623 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6e097c5-cc31-4829-888e-3837adf23172-config-out\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664653 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664774 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664728 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664808 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664772 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664808 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664796 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664812 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e097c5-cc31-4829-888e-3837adf23172-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664833 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6e097c5-cc31-4829-888e-3837adf23172-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664870 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-web-config\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664890 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.664988 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.664935 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4qr\" (UniqueName: \"kubernetes.io/projected/c6e097c5-cc31-4829-888e-3837adf23172-kube-api-access-qf4qr\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.677475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677445 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 01:16:11.677475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677445 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 01:16:11.677657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677612 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 01:16:11.677657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677630 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 01:16:11.677657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677632 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 01:16:11.677657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677643 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bstbs\"" Nov 02 01:16:11.677657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677648 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 01:16:11.677892 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.677704 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 01:16:11.766198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766099 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766169 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766197 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766222 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e097c5-cc31-4829-888e-3837adf23172-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766242 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6e097c5-cc31-4829-888e-3837adf23172-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766272 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-web-config\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766296 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766318 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4qr\" (UniqueName: \"kubernetes.io/projected/c6e097c5-cc31-4829-888e-3837adf23172-kube-api-access-qf4qr\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766345 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6e097c5-cc31-4829-888e-3837adf23172-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766381 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766420 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6e097c5-cc31-4829-888e-3837adf23172-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766453 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6e097c5-cc31-4829-888e-3837adf23172-config-out\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.766888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.766483 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.767326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.767298 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e097c5-cc31-4829-888e-3837adf23172-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.767999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.767929 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6e097c5-cc31-4829-888e-3837adf23172-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.768380 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.768356 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6e097c5-cc31-4829-888e-3837adf23172-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.769523 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.769477 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770104 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770142 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770104 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770179 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770194 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6e097c5-cc31-4829-888e-3837adf23172-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770690 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770670 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.770953 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.770937 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6e097c5-cc31-4829-888e-3837adf23172-config-out\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.771336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.771314 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6e097c5-cc31-4829-888e-3837adf23172-web-config\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.779814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.779792 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4qr\" (UniqueName: \"kubernetes.io/projected/c6e097c5-cc31-4829-888e-3837adf23172-kube-api-access-qf4qr\") pod \"alertmanager-main-0\" (UID: \"c6e097c5-cc31-4829-888e-3837adf23172\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:11.964763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:11.964719 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 01:16:12.045259 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:12.045225 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e5a629-74d2-4e37-9867-cb9bd4f9bea9" path="/var/lib/kubelet/pods/16e5a629-74d2-4e37-9867-cb9bd4f9bea9/volumes" Nov 02 01:16:12.045974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:12.045730 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c9abb4-26a9-469b-b161-2bde675fc7ba" path="/var/lib/kubelet/pods/c5c9abb4-26a9-469b-b161-2bde675fc7ba/volumes" Nov 02 01:16:12.279530 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:12.278392 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 01:16:12.282293 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:16:12.282266 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e097c5_cc31_4829_888e_3837adf23172.slice/crio-e51547a51637fcf34b00481d12ad770980fd39aba3bd8e12c33aab32ca496e4e WatchSource:0}: Error finding container e51547a51637fcf34b00481d12ad770980fd39aba3bd8e12c33aab32ca496e4e: Status 404 returned error can't find the container with id e51547a51637fcf34b00481d12ad770980fd39aba3bd8e12c33aab32ca496e4e Nov 02 01:16:12.477000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:12.476971 4552 generic.go:356] "Generic (PLEG): container finished" podID="c6e097c5-cc31-4829-888e-3837adf23172" containerID="cc0401db5275f1d206ed58f828684fd7275ba72dd00e515569ac054a969b62a7" exitCode=0 Nov 02 01:16:12.477423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:12.477046 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerDied","Data":"cc0401db5275f1d206ed58f828684fd7275ba72dd00e515569ac054a969b62a7"} Nov 02 01:16:12.477423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:12.477108 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"e51547a51637fcf34b00481d12ad770980fd39aba3bd8e12c33aab32ca496e4e"} Nov 02 01:16:13.486549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.486513 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"229555ebf42551335161efbfee6276272a4b63e38b4aa94246740edadfc247b8"} Nov 02 01:16:13.486985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.486589 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"de790f479946df7508d6aad51c65f7b1d7d146f65656f9e1b5f58aafacb340eb"} Nov 02 01:16:13.486985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.486653 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"2d1d6b53aed5eaa1391f020c63a118089b5040d8fbe76a96f97ec4d108a2939b"} Nov 02 01:16:13.486985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.486693 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"b5b5db88fb08144cec16d32fa77b44826ac0e8aa33b41cdbfd64cf6c70eeaf38"} Nov 02 01:16:13.486985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.486740 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"4732f1188489553786f4a1e8a93d189274122329239725e69bf65f56f16ea5c6"} Nov 02 01:16:13.486985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.486775 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6e097c5-cc31-4829-888e-3837adf23172","Type":"ContainerStarted","Data":"7e012e48641957dcc7dadfb3a3f8b5537957c890d9839768cc4dfcaeec986a28"} Nov 02 01:16:13.521433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:13.521352 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.50901087 podStartE2EDuration="2.521334941s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.324071ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:16:13.519599757 +0000 UTC m=+73.625388708" watchObservedRunningTime="2026-11-02 01:16:13.521334941 +0000 UTC m=+73.627123859" Nov 02 01:16:16.703014 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:16.702978 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-74jqn-trlw4"] Nov 02 01:16:16.703559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:16.703209 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-74jqn-trlw4" podUID="844ca99b-cdd2-4659-bd4d-839c69e15f46" containerName="insights-gathering" containerID="cri-o://4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c" gracePeriod=30 Nov 02 01:16:17.000165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.000106 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:16:17.116016 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.115975 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") pod \"844ca99b-cdd2-4659-bd4d-839c69e15f46\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " Nov 02 01:16:17.116243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.116045 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") pod \"844ca99b-cdd2-4659-bd4d-839c69e15f46\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " Nov 02 01:16:17.116243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.116159 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") pod \"844ca99b-cdd2-4659-bd4d-839c69e15f46\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " Nov 02 01:16:17.116243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.116223 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") pod \"844ca99b-cdd2-4659-bd4d-839c69e15f46\" (UID: \"844ca99b-cdd2-4659-bd4d-839c69e15f46\") " Nov 02 01:16:17.116466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.116436 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle" pod "844ca99b-cdd2-4659-bd4d-839c69e15f46" (UID: "844ca99b-cdd2-4659-bd4d-839c69e15f46"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:17.116510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.116463 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path" pod "844ca99b-cdd2-4659-bd4d-839c69e15f46" (UID: "844ca99b-cdd2-4659-bd4d-839c69e15f46"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:17.116543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.116533 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle" pod "844ca99b-cdd2-4659-bd4d-839c69e15f46" (UID: "844ca99b-cdd2-4659-bd4d-839c69e15f46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:17.118692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.118667 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw" pod "844ca99b-cdd2-4659-bd4d-839c69e15f46" (UID: "844ca99b-cdd2-4659-bd4d-839c69e15f46"). InnerVolumeSpecName "kube-api-access-4ztjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:17.217332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.217302 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ztjw\" (UniqueName: \"kubernetes.io/projected/844ca99b-cdd2-4659-bd4d-839c69e15f46-kube-api-access-4ztjw\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:17.217332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.217331 4552 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-service-ca-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:17.217510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.217342 4552 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/844ca99b-cdd2-4659-bd4d-839c69e15f46-archives-path\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:17.217510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.217354 4552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/844ca99b-cdd2-4659-bd4d-839c69e15f46-trusted-ca-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:17.503956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.503867 4552 generic.go:356] "Generic (PLEG): container finished" podID="844ca99b-cdd2-4659-bd4d-839c69e15f46" containerID="4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c" exitCode=2 Nov 02 01:16:17.503956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.503913 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-74jqn-trlw4" Nov 02 01:16:17.503956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.503951 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-74jqn-trlw4" event={"ID":"844ca99b-cdd2-4659-bd4d-839c69e15f46","Type":"ContainerDied","Data":"4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c"} Nov 02 01:16:17.504201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.503996 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-74jqn-trlw4" event={"ID":"844ca99b-cdd2-4659-bd4d-839c69e15f46","Type":"ContainerDied","Data":"145a835ff7994fe37c109461fbc79db7247d27b5bd72dabdaf448d44f0f84fb2"} Nov 02 01:16:17.504201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.504017 4552 scope.go:122] "RemoveContainer" containerID="4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c" Nov 02 01:16:17.512451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.512430 4552 scope.go:122] "RemoveContainer" containerID="4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c" Nov 02 01:16:17.512717 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:16:17.512702 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c\": container with ID starting with 4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c not found: ID does not exist" containerID="4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c" Nov 02 01:16:17.512766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.512725 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c"} err="failed to get container status \"4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c\": rpc error: code = NotFound desc = could not find container \"4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c\": container with ID starting with 4b23d16bfb1f41489f8404fe61ae35efcb2148ec93e449750219038ce9a8974c not found: ID does not exist" Nov 02 01:16:17.525004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.524981 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-74jqn-trlw4"] Nov 02 01:16:17.528648 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:17.528626 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-74jqn-trlw4"] Nov 02 01:16:18.041316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:18.041285 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844ca99b-cdd2-4659-bd4d-839c69e15f46" path="/var/lib/kubelet/pods/844ca99b-cdd2-4659-bd4d-839c69e15f46/volumes" Nov 02 01:16:18.766473 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:18.766439 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:18.766672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:18.766503 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:18.771070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:18.771048 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:19.516636 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:19.516607 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c8fb7895f-dxl9d" Nov 02 01:16:19.565382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:19.565351 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7585fddb94-8ktbt"] Nov 02 01:16:27.749036 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:16:27.748999 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:16:44.585365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.585308 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7585fddb94-8ktbt" podUID="09af22f1-5a55-4e91-8ec4-32bd81ac002c" containerName="console" containerID="cri-o://ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a" gracePeriod=15 Nov 02 01:16:44.830513 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.830489 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:16:44.957628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957548 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.957628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957599 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.957628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957620 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.957909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957830 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.957909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957883 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.958020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957952 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.958020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.957988 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4\" (UniqueName: \"kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.958139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958030 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config\") pod \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\" (UID: \"09af22f1-5a55-4e91-8ec4-32bd81ac002c\") " Nov 02 01:16:44.958139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958061 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:44.958270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958242 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:44.958270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958164 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:44.958373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958258 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:44.958373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958311 4552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-trusted-ca-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:44.958559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.958535 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:44.960252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.960224 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "kube-api-access-4vrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:44.960569 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.960547 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:44.960569 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:44.960560 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config" pod "09af22f1-5a55-4e91-8ec4-32bd81ac002c" (UID: "09af22f1-5a55-4e91-8ec4-32bd81ac002c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:45.059013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.058984 4552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-serving-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.059013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.059016 4552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-oauth-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.059224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.059026 4552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-oauth-serving-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.059224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.059036 4552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.059224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.059044 4552 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09af22f1-5a55-4e91-8ec4-32bd81ac002c-tmp\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.059224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.059054 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vrw4\" (UniqueName: \"kubernetes.io/projected/09af22f1-5a55-4e91-8ec4-32bd81ac002c-kube-api-access-4vrw4\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.059224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.059063 4552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09af22f1-5a55-4e91-8ec4-32bd81ac002c-console-config\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:45.633969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.629636 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585fddb94-8ktbt" event={"ID":"09af22f1-5a55-4e91-8ec4-32bd81ac002c","Type":"ContainerDied","Data":"ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a"} Nov 02 01:16:45.633969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.629601 4552 generic.go:356] "Generic (PLEG): container finished" podID="09af22f1-5a55-4e91-8ec4-32bd81ac002c" containerID="ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a" exitCode=0 Nov 02 01:16:45.633969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.629740 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585fddb94-8ktbt" event={"ID":"09af22f1-5a55-4e91-8ec4-32bd81ac002c","Type":"ContainerDied","Data":"2c986fa6e0f9b4f3193045f738f451e1b3f68d1d5b127251d47c9d162ae628ab"} Nov 02 01:16:45.633969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.629770 4552 scope.go:122] "RemoveContainer" containerID="ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a" Nov 02 01:16:45.633969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.629804 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585fddb94-8ktbt" Nov 02 01:16:45.643272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.643245 4552 scope.go:122] "RemoveContainer" containerID="ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a" Nov 02 01:16:45.643582 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:16:45.643563 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a\": container with ID starting with ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a not found: ID does not exist" containerID="ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a" Nov 02 01:16:45.643652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.643596 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a"} err="failed to get container status \"ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a\": rpc error: code = NotFound desc = could not find container \"ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a\": container with ID starting with ce36bbbaa18d4aa15bd5cbda7e68dbe9c491ceebc7fc376876549aa08c6fa98a not found: ID does not exist" Nov 02 01:16:45.654431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.654406 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7585fddb94-8ktbt"] Nov 02 01:16:45.658700 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:45.658673 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7585fddb94-8ktbt"] Nov 02 01:16:46.040882 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:16:46.040846 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09af22f1-5a55-4e91-8ec4-32bd81ac002c" path="/var/lib/kubelet/pods/09af22f1-5a55-4e91-8ec4-32bd81ac002c/volumes" Nov 02 01:16:57.748548 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:16:57.748511 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:17:27.748513 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:17:27.748473 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:17:57.748327 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:17:57.748283 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:18:27.749688 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:18:27.749643 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:18:47.528463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528377 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs"] Nov 02 01:18:47.529021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528832 4552 cpu_manager.go:414] "containerMap: removing container" podUID="844ca99b-cdd2-4659-bd4d-839c69e15f46" containerName="insights-gathering" Nov 02 01:18:47.529021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528852 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844ca99b-cdd2-4659-bd4d-839c69e15f46" containerName="insights-gathering" Nov 02 01:18:47.529021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528870 4552 cpu_manager.go:414] "containerMap: removing container" podUID="09af22f1-5a55-4e91-8ec4-32bd81ac002c" containerName="console" Nov 02 01:18:47.529021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528878 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09af22f1-5a55-4e91-8ec4-32bd81ac002c" containerName="console" Nov 02 01:18:47.529021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528987 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="844ca99b-cdd2-4659-bd4d-839c69e15f46" containerName="insights-gathering" Nov 02 01:18:47.529021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.528999 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="09af22f1-5a55-4e91-8ec4-32bd81ac002c" containerName="console" Nov 02 01:18:47.530979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.530957 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.535835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.535811 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 01:18:47.535946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.535867 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 01:18:47.536058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.536041 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v82s5\"" Nov 02 01:18:47.540924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.540899 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs"] Nov 02 01:18:47.622247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.622221 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn"] Nov 02 01:18:47.624516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.624501 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.628094 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.628074 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-drm45\"" Nov 02 01:18:47.635650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.635627 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn"] Nov 02 01:18:47.642871 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.642850 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mht\" (UniqueName: \"kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.642989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.642891 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.642989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.642972 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.744240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744209 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.744414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744252 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.744414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744275 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.744414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744352 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.744414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744382 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkx72\" (UniqueName: \"kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.744608 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744416 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mht\" (UniqueName: \"kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.744679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744656 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.744735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.744665 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.748259 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.748232 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb"] Nov 02 01:18:47.750623 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.750604 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.761895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.761878 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-6mldh\"" Nov 02 01:18:47.775832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.775798 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb"] Nov 02 01:18:47.776612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.776591 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mht\" (UniqueName: \"kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.834618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.834552 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6"] Nov 02 01:18:47.837650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.837635 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:47.844334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.844309 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:18:47.844856 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.844826 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.844967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.844890 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkx72\" (UniqueName: \"kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.844967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.844934 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.845070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.844977 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.845070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.845007 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qmmg\" (UniqueName: \"kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.845206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.845170 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.845280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.845257 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.845559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.845538 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.847723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.847699 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6"] Nov 02 01:18:47.857013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.856989 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkx72\" (UniqueName: \"kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.861703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.861684 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bpxjq\"" Nov 02 01:18:47.937473 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.937442 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:18:47.945583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.945559 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.945711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.945589 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qmmg\" (UniqueName: \"kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.945711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.945616 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:47.945711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.945650 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:47.945711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.945678 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvgv\" (UniqueName: \"kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:47.945711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.945703 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.946050 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.946024 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.946239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.946221 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:47.958401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:47.958381 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qmmg\" (UniqueName: \"kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:48.046694 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.046663 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvgv\" (UniqueName: \"kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.046874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.046724 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.046874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.046765 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.047078 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.047051 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.047170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.047093 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.062637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.062616 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:18:48.062915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.062898 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvgv\" (UniqueName: \"kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.126411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.126366 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs"] Nov 02 01:18:48.129087 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:48.129053 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d22a26b_d71e_4005_9c4d_2e6a59ab9a4b.slice/crio-88e2da7b59f51cdb2ae28a372aa059cea482db68ac436f791be0231235414ea2 WatchSource:0}: Error finding container 88e2da7b59f51cdb2ae28a372aa059cea482db68ac436f791be0231235414ea2: Status 404 returned error can't find the container with id 88e2da7b59f51cdb2ae28a372aa059cea482db68ac436f791be0231235414ea2 Nov 02 01:18:48.153978 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.153951 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:18:48.210819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.210788 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn"] Nov 02 01:18:48.215238 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:48.215210 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59c6a57_e714_4ba4_a553_bffb240b26bb.slice/crio-0a67e36716ccc356ddf7eb7d123ea768e4f1301aebd510145d09db93fca9b77c WatchSource:0}: Error finding container 0a67e36716ccc356ddf7eb7d123ea768e4f1301aebd510145d09db93fca9b77c: Status 404 returned error can't find the container with id 0a67e36716ccc356ddf7eb7d123ea768e4f1301aebd510145d09db93fca9b77c Nov 02 01:18:48.335061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.335031 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb"] Nov 02 01:18:48.337812 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:48.337784 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db39889_40f5_4ad2_9fae_22f28e342c75.slice/crio-cb31ba228fbd59e9d758997b86263678623f5e31c84df6a3fcda2886731d8db7 WatchSource:0}: Error finding container cb31ba228fbd59e9d758997b86263678623f5e31c84df6a3fcda2886731d8db7: Status 404 returned error can't find the container with id cb31ba228fbd59e9d758997b86263678623f5e31c84df6a3fcda2886731d8db7 Nov 02 01:18:48.430584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.430563 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6"] Nov 02 01:18:48.432820 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:48.432793 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f34e9cf_87c2_426c_8915_55f5a9dd0fea.slice/crio-f854e31655847ef0b42a4c9c02cf4fc01560d1af5170ebf296dae3275da2ebdf WatchSource:0}: Error finding container f854e31655847ef0b42a4c9c02cf4fc01560d1af5170ebf296dae3275da2ebdf: Status 404 returned error can't find the container with id f854e31655847ef0b42a4c9c02cf4fc01560d1af5170ebf296dae3275da2ebdf Nov 02 01:18:48.726533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.726505 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq"] Nov 02 01:18:48.729001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.728975 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.732694 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.732674 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7wwph\"" Nov 02 01:18:48.739079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.739057 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq"] Nov 02 01:18:48.853057 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.853022 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.853226 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.853184 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5s5w\" (UniqueName: \"kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.853268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.853228 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.954505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.954460 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.954756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.954530 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.954756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.954633 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5s5w\" (UniqueName: \"kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.955413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.955387 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.955680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.955660 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:48.970910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:48.970881 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5s5w\" (UniqueName: \"kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:49.042564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.042486 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:18:49.066591 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.066476 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" event={"ID":"3db39889-40f5-4ad2-9fae-22f28e342c75","Type":"ContainerStarted","Data":"cb31ba228fbd59e9d758997b86263678623f5e31c84df6a3fcda2886731d8db7"} Nov 02 01:18:49.069483 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.069376 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" event={"ID":"d59c6a57-e714-4ba4-a553-bffb240b26bb","Type":"ContainerStarted","Data":"0a67e36716ccc356ddf7eb7d123ea768e4f1301aebd510145d09db93fca9b77c"} Nov 02 01:18:49.070993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.070923 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" event={"ID":"0f34e9cf-87c2-426c-8915-55f5a9dd0fea","Type":"ContainerStarted","Data":"f854e31655847ef0b42a4c9c02cf4fc01560d1af5170ebf296dae3275da2ebdf"} Nov 02 01:18:49.073843 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.073804 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" event={"ID":"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b","Type":"ContainerStarted","Data":"88e2da7b59f51cdb2ae28a372aa059cea482db68ac436f791be0231235414ea2"} Nov 02 01:18:49.339227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.338469 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq"] Nov 02 01:18:49.341928 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:49.341893 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826866da_0827_4c60_b767_895198c5e9da.slice/crio-d74418e4be15b9bfcd4da3a6fef44356a02b2238036746e382aa8eb1a7619126 WatchSource:0}: Error finding container d74418e4be15b9bfcd4da3a6fef44356a02b2238036746e382aa8eb1a7619126: Status 404 returned error can't find the container with id d74418e4be15b9bfcd4da3a6fef44356a02b2238036746e382aa8eb1a7619126 Nov 02 01:18:49.576132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.576082 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5"] Nov 02 01:18:49.579689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.579665 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.586431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.586405 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-m9z6d\"" Nov 02 01:18:49.589593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.589520 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5"] Nov 02 01:18:49.662273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.662234 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.662449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.662319 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.662449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.662363 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlktm\" (UniqueName: \"kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.763770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.763714 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.764283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.763774 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.764283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.764172 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.764283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.764233 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlktm\" (UniqueName: \"kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.764454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.764352 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.778475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.778447 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlktm\" (UniqueName: \"kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:49.898705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:49.898394 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:18:50.080243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:50.080203 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" event={"ID":"826866da-0827-4c60-b767-895198c5e9da","Type":"ContainerStarted","Data":"d74418e4be15b9bfcd4da3a6fef44356a02b2238036746e382aa8eb1a7619126"} Nov 02 01:18:50.196718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:50.196644 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5"] Nov 02 01:18:50.200949 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:50.200910 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03882b08_3940_4e35_8eb2_2fc44b83cb90.slice/crio-824aef69930aae783e84b5a10b2bb83ba2df9afd78c34ec336d61ce83008559f WatchSource:0}: Error finding container 824aef69930aae783e84b5a10b2bb83ba2df9afd78c34ec336d61ce83008559f: Status 404 returned error can't find the container with id 824aef69930aae783e84b5a10b2bb83ba2df9afd78c34ec336d61ce83008559f Nov 02 01:18:51.087419 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:51.087330 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" event={"ID":"03882b08-3940-4e35-8eb2-2fc44b83cb90","Type":"ContainerStarted","Data":"824aef69930aae783e84b5a10b2bb83ba2df9afd78c34ec336d61ce83008559f"} Nov 02 01:18:52.562178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.562140 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg"] Nov 02 01:18:52.565354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.565332 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.595448 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.595419 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nh7nl\"" Nov 02 01:18:52.599780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.599745 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg"] Nov 02 01:18:52.692470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.692111 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.692470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.692206 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.692470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.692254 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xt7\" (UniqueName: \"kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.794358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.793358 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.794358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.793440 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.794358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.793469 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xt7\" (UniqueName: \"kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.795237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.794968 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.795237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.795184 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.813264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.813192 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xt7\" (UniqueName: \"kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.837525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.837498 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc"] Nov 02 01:18:52.840945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.840133 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.857175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.855971 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 01:18:52.857175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.856053 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 01:18:52.857175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.856204 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-48jtq\"" Nov 02 01:18:52.857175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.856247 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 01:18:52.861936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.861908 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc"] Nov 02 01:18:52.884658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.884324 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:18:52.922951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.920702 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9"] Nov 02 01:18:52.924846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.924824 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.939962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.939134 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9"] Nov 02 01:18:52.948504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.948387 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-wfnp9\"" Nov 02 01:18:52.995699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.995590 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.995889 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.995731 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.995962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.995886 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.996020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.995917 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996046 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.996081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996070 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996197 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2sg\" (UniqueName: \"kubernetes.io/projected/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-kube-api-access-2n2sg\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996244 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996307 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996384 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996419 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.996700 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996500 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.996847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996788 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.996926 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996854 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.996926 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996878 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.997029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996931 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:52.997029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996948 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:52.997029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:52.996968 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qkv\" (UniqueName: \"kubernetes.io/projected/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-kube-api-access-26qkv\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.097417 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097380 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097426 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097457 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097482 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097529 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097554 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097587 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2sg\" (UniqueName: \"kubernetes.io/projected/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-kube-api-access-2n2sg\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097616 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097646 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097675 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.097650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097703 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097725 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097759 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097799 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097838 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097870 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097896 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.098355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.097933 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qkv\" (UniqueName: \"kubernetes.io/projected/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-kube-api-access-26qkv\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.098777 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.098606 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.099297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.099134 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.099401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.099375 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.099401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.099388 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.100658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.099621 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.100658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.100093 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.100658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.100129 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.100658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.100352 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.100658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.100565 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.100969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.100745 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.103758 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.103735 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.108432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.108406 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.108789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.108765 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.112675 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.112648 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qkv\" (UniqueName: \"kubernetes.io/projected/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-kube-api-access-26qkv\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.115899 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.115584 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f2a834-c86c-4af6-bbde-17ba9e356e2d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4nkxc\" (UID: \"a9f2a834-c86c-4af6-bbde-17ba9e356e2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.116004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.115987 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.117406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.117335 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2sg\" (UniqueName: \"kubernetes.io/projected/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-kube-api-access-2n2sg\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.128753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.128702 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3b5284-e92a-42aa-95b0-0fe67d121bd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9\" (UID: \"3f3b5284-e92a-42aa-95b0-0fe67d121bd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.168283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.168253 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:18:53.235327 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.234641 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg"] Nov 02 01:18:53.241332 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:53.241281 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a0716b_f78f_41b9_b18d_e2c97cbe8ba3.slice/crio-827c31738fc5c3535d6cd7a776cb6d848a652a6272acba18fe0445cf0edf2a46 WatchSource:0}: Error finding container 827c31738fc5c3535d6cd7a776cb6d848a652a6272acba18fe0445cf0edf2a46: Status 404 returned error can't find the container with id 827c31738fc5c3535d6cd7a776cb6d848a652a6272acba18fe0445cf0edf2a46 Nov 02 01:18:53.245582 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.245558 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:18:53.470531 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.470447 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc"] Nov 02 01:18:53.473886 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:53.473852 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f2a834_c86c_4af6_bbde_17ba9e356e2d.slice/crio-a9b319cf8ee636585a95672ada0a27db295a4b380212b0d8216a8ee9bc55949a WatchSource:0}: Error finding container a9b319cf8ee636585a95672ada0a27db295a4b380212b0d8216a8ee9bc55949a: Status 404 returned error can't find the container with id a9b319cf8ee636585a95672ada0a27db295a4b380212b0d8216a8ee9bc55949a Nov 02 01:18:53.585022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:53.584998 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9"] Nov 02 01:18:53.591648 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:53.591588 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3b5284_e92a_42aa_95b0_0fe67d121bd7.slice/crio-c66b836c12b28fe0bc2d980e44e6768f9f775ae2df47663c927b01b1ce7c8498 WatchSource:0}: Error finding container c66b836c12b28fe0bc2d980e44e6768f9f775ae2df47663c927b01b1ce7c8498: Status 404 returned error can't find the container with id c66b836c12b28fe0bc2d980e44e6768f9f775ae2df47663c927b01b1ce7c8498 Nov 02 01:18:54.119264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:54.119224 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" event={"ID":"3f3b5284-e92a-42aa-95b0-0fe67d121bd7","Type":"ContainerStarted","Data":"c66b836c12b28fe0bc2d980e44e6768f9f775ae2df47663c927b01b1ce7c8498"} Nov 02 01:18:54.136200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:54.127331 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" event={"ID":"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3","Type":"ContainerStarted","Data":"827c31738fc5c3535d6cd7a776cb6d848a652a6272acba18fe0445cf0edf2a46"} Nov 02 01:18:54.144191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:54.144079 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" event={"ID":"a9f2a834-c86c-4af6-bbde-17ba9e356e2d","Type":"ContainerStarted","Data":"a9b319cf8ee636585a95672ada0a27db295a4b380212b0d8216a8ee9bc55949a"} Nov 02 01:18:55.390208 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.390152 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg"] Nov 02 01:18:55.396907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.396476 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.403579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.402682 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg"] Nov 02 01:18:55.418961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.417813 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4"] Nov 02 01:18:55.422155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.421833 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.434478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.434455 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4"] Nov 02 01:18:55.451320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.449920 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf"] Nov 02 01:18:55.456814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.456691 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.463699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.463653 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf"] Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523418 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscbb\" (UniqueName: \"kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523468 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523537 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523568 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc765\" (UniqueName: \"kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523594 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523667 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523816 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523850 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfplh\" (UniqueName: \"kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.524008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.523885 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625358 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625430 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc765\" (UniqueName: \"kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625459 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625571 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625606 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625650 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfplh\" (UniqueName: \"kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625695 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625753 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscbb\" (UniqueName: \"kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625780 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.625803 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.626195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.626045 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.626864 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.626285 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.627567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.627005 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.627567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.627284 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.627702 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.627643 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.644589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.644566 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc765\" (UniqueName: \"kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.645491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.645443 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscbb\" (UniqueName: \"kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:55.647384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.647341 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfplh\" (UniqueName: \"kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.723967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.723591 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:18:55.745470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.745264 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:18:55.791168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:55.790903 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:18:56.110872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:56.110810 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4"] Nov 02 01:18:56.115807 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:56.115687 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg"] Nov 02 01:18:56.120688 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:56.120661 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb868b2f8_9412_459a_8c5d_c28617e5e341.slice/crio-e4484b20e48a4d3cb9a936460600ff088968f4a12c07732f6d27e3300fa5a226 WatchSource:0}: Error finding container e4484b20e48a4d3cb9a936460600ff088968f4a12c07732f6d27e3300fa5a226: Status 404 returned error can't find the container with id e4484b20e48a4d3cb9a936460600ff088968f4a12c07732f6d27e3300fa5a226 Nov 02 01:18:56.181516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:56.181480 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" event={"ID":"b868b2f8-9412-459a-8c5d-c28617e5e341","Type":"ContainerStarted","Data":"e4484b20e48a4d3cb9a936460600ff088968f4a12c07732f6d27e3300fa5a226"} Nov 02 01:18:56.187969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:56.187938 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf"] Nov 02 01:18:56.195278 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:18:56.195247 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1bbcb8_4880_4150_9c9e_610a1b1535c0.slice/crio-3058eeed11b6675f6b644b09768522df981803328acefe2617c5b5eed4d9a868 WatchSource:0}: Error finding container 3058eeed11b6675f6b644b09768522df981803328acefe2617c5b5eed4d9a868: Status 404 returned error can't find the container with id 3058eeed11b6675f6b644b09768522df981803328acefe2617c5b5eed4d9a868 Nov 02 01:18:56.197689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:56.197659 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" event={"ID":"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44","Type":"ContainerStarted","Data":"436ecc34fa4fec81aebd67ec83dfd94663d78761ca3602a15094e0d307094470"} Nov 02 01:18:57.213325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:18:57.213269 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" event={"ID":"af1bbcb8-4880-4150-9c9e-610a1b1535c0","Type":"ContainerStarted","Data":"3058eeed11b6675f6b644b09768522df981803328acefe2617c5b5eed4d9a868"} Nov 02 01:18:57.812516 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:18:57.812470 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:19:03.526695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:03.525856 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:19:03.526695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:03.525961 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:19:03.526695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:03.526019 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:19:04.322698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:04.321747 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" event={"ID":"a9f2a834-c86c-4af6-bbde-17ba9e356e2d","Type":"ContainerStarted","Data":"f0efbb0afd5f91e982642bc8edf075c92c3626c3ad7edf9ef0eb1dee84961ad1"} Nov 02 01:19:04.663782 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:04.663741 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:19:04.664168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:04.663818 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:19:04.664168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:04.663856 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:19:05.170255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.168872 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:19:05.175435 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.175160 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:19:05.200419 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.200218 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podStartSLOduration=3.153688638 podStartE2EDuration="13.200199587s" totalImagesPullingTime="10.046510949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:04.349919906 +0000 UTC m=+244.455708822" watchObservedRunningTime="2026-11-02 01:19:05.200199587 +0000 UTC m=+245.305988501" Nov 02 01:19:05.341964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.341839 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" event={"ID":"3f3b5284-e92a-42aa-95b0-0fe67d121bd7","Type":"ContainerStarted","Data":"e111045a225a5cae88602cdea8205d4158aad5817cccff5ab571fcc789a8d8c5"} Nov 02 01:19:05.341964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.341902 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:19:05.343577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.343519 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:19:05.401261 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:05.398998 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" podStartSLOduration=2.33237552 podStartE2EDuration="13.398977071s" totalImagesPullingTime="11.066601551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:05.367995532 +0000 UTC m=+245.473784446" watchObservedRunningTime="2026-11-02 01:19:05.398977071 +0000 UTC m=+245.504765987" Nov 02 01:19:06.246890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:06.246842 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:19:06.253144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:06.253102 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:19:06.348034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:06.348008 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:19:06.350494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:06.350256 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9" Nov 02 01:19:08.370233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:08.370156 4552 generic.go:356] "Generic (PLEG): container finished" podID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerID="bb411a52767e24bd2faaa22827051e0be07477bff464a9ecc993944582062ada" exitCode=0 Nov 02 01:19:08.370632 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:08.370302 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" event={"ID":"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b","Type":"ContainerDied","Data":"bb411a52767e24bd2faaa22827051e0be07477bff464a9ecc993944582062ada"} Nov 02 01:19:08.376085 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:08.374975 4552 generic.go:356] "Generic (PLEG): container finished" podID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerID="160c2a1a0a2eb0bd200b7f57dfb5224603089db1a44f5559bacb9d6da34a7c77" exitCode=0 Nov 02 01:19:08.376085 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:08.376000 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" event={"ID":"0f34e9cf-87c2-426c-8915-55f5a9dd0fea","Type":"ContainerDied","Data":"160c2a1a0a2eb0bd200b7f57dfb5224603089db1a44f5559bacb9d6da34a7c77"} Nov 02 01:19:09.382452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:09.382421 4552 generic.go:356] "Generic (PLEG): container finished" podID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerID="0f52ca0c4784a819b6db9674ba6cec17fd96d4dcaf718198306cac0ba8301ac1" exitCode=0 Nov 02 01:19:09.382815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:09.382518 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" event={"ID":"af1bbcb8-4880-4150-9c9e-610a1b1535c0","Type":"ContainerDied","Data":"0f52ca0c4784a819b6db9674ba6cec17fd96d4dcaf718198306cac0ba8301ac1"} Nov 02 01:19:09.384330 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:09.384303 4552 generic.go:356] "Generic (PLEG): container finished" podID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerID="61d3546c5f44baf628bdd3006540e8a862a3365691a399c7a3cef09ef954c8e9" exitCode=0 Nov 02 01:19:09.384451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:09.384423 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" event={"ID":"03882b08-3940-4e35-8eb2-2fc44b83cb90","Type":"ContainerDied","Data":"61d3546c5f44baf628bdd3006540e8a862a3365691a399c7a3cef09ef954c8e9"} Nov 02 01:19:09.387863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:09.387838 4552 generic.go:356] "Generic (PLEG): container finished" podID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerID="12d991353fc8ded9dc78c88f960786a6155f25fba172ef59e150757776b48e3b" exitCode=0 Nov 02 01:19:09.387965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:09.387881 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" event={"ID":"d59c6a57-e714-4ba4-a553-bffb240b26bb","Type":"ContainerDied","Data":"12d991353fc8ded9dc78c88f960786a6155f25fba172ef59e150757776b48e3b"} Nov 02 01:19:10.395525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.395446 4552 generic.go:356] "Generic (PLEG): container finished" podID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerID="4ee236c1d842e8a178f1f59e2bd435634271c87c5822958bd0fb0573ca95327a" exitCode=0 Nov 02 01:19:10.396061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.395528 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" event={"ID":"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b","Type":"ContainerDied","Data":"4ee236c1d842e8a178f1f59e2bd435634271c87c5822958bd0fb0573ca95327a"} Nov 02 01:19:10.397253 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.397227 4552 generic.go:356] "Generic (PLEG): container finished" podID="826866da-0827-4c60-b767-895198c5e9da" containerID="0a47de83678f96f09047a7722b38f28114a1b4612e9f2288fd0d41039931e7d8" exitCode=0 Nov 02 01:19:10.397370 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.397306 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" event={"ID":"826866da-0827-4c60-b767-895198c5e9da","Type":"ContainerDied","Data":"0a47de83678f96f09047a7722b38f28114a1b4612e9f2288fd0d41039931e7d8"} Nov 02 01:19:10.400058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.400030 4552 generic.go:356] "Generic (PLEG): container finished" podID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerID="14ef8c5607e2c71118fb0503956815bcb292d33387511421975ad71954a1225c" exitCode=0 Nov 02 01:19:10.400194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.400163 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" event={"ID":"3db39889-40f5-4ad2-9fae-22f28e342c75","Type":"ContainerDied","Data":"14ef8c5607e2c71118fb0503956815bcb292d33387511421975ad71954a1225c"} Nov 02 01:19:10.404307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.404281 4552 generic.go:356] "Generic (PLEG): container finished" podID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerID="c6f3ad083a979c4388e60bbae436ee0a04ff507d40c24fcbf1492071b6e97e42" exitCode=0 Nov 02 01:19:10.404430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.404365 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" event={"ID":"0f34e9cf-87c2-426c-8915-55f5a9dd0fea","Type":"ContainerDied","Data":"c6f3ad083a979c4388e60bbae436ee0a04ff507d40c24fcbf1492071b6e97e42"} Nov 02 01:19:10.406286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.406262 4552 generic.go:356] "Generic (PLEG): container finished" podID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerID="cb41ebc777a739dc6f7f2a491aa71e30644f2af1306da4b3422f45ee812de919" exitCode=0 Nov 02 01:19:10.406392 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.406343 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" event={"ID":"b868b2f8-9412-459a-8c5d-c28617e5e341","Type":"ContainerDied","Data":"cb41ebc777a739dc6f7f2a491aa71e30644f2af1306da4b3422f45ee812de919"} Nov 02 01:19:10.408166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.408047 4552 generic.go:356] "Generic (PLEG): container finished" podID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerID="4c1f18af5cc350ba390e4a5cb8f130d79e9b3712d0a775f314843f09c4373f7f" exitCode=0 Nov 02 01:19:10.408166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:10.408105 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" event={"ID":"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44","Type":"ContainerDied","Data":"4c1f18af5cc350ba390e4a5cb8f130d79e9b3712d0a775f314843f09c4373f7f"} Nov 02 01:19:10.855946 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:19:10.855887 4552 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/79/79d625778395baede93cada33312fd4ec49f3801c9d9c3bbd6a064bf67fd5f89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T011909Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=144ba4dfc8f3a0ce0a655b6068fba99b7376fb9de4005fff7a9c117d351eeb6e®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=rhcl-1----rhcl-operator-bundle&akamai_signature=exp=1793583249~hmac=d214117b4c77ef0abfec8b00d81fea7863993194ae7300f784a09a091708bb77\": remote error: tls: internal error" image="registry.redhat.io/rhcl-1/rhcl-operator-bundle@sha256:8b1a5c448365b2b2ca6ddfbc5a584341be4a582170e0817aa400442700cc951e" Nov 02 01:19:10.856239 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:19:10.856199 4552 kuberuntime_manager.go:1707] "Unhandled Error" err="init container pull start failed in pod 6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn_openshift-marketplace(d59c6a57-e714-4ba4-a553-bffb240b26bb): ErrImagePull: unable to pull image: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/79/79d625778395baede93cada33312fd4ec49f3801c9d9c3bbd6a064bf67fd5f89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T011909Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=144ba4dfc8f3a0ce0a655b6068fba99b7376fb9de4005fff7a9c117d351eeb6e®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=rhcl-1----rhcl-operator-bundle&akamai_signature=exp=1793583249~hmac=d214117b4c77ef0abfec8b00d81fea7863993194ae7300f784a09a091708bb77\": remote error: tls: internal error" logger="UnhandledError" Nov 02 01:19:10.856342 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:19:10.856273 4552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ErrImagePull: \"unable to pull image: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/79/79d625778395baede93cada33312fd4ec49f3801c9d9c3bbd6a064bf67fd5f89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261102%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261102T011909Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=144ba4dfc8f3a0ce0a655b6068fba99b7376fb9de4005fff7a9c117d351eeb6e®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=rhcl-1----rhcl-operator-bundle&akamai_signature=exp=1793583249~hmac=d214117b4c77ef0abfec8b00d81fea7863993194ae7300f784a09a091708bb77\\\": remote error: tls: internal error\"" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" Nov 02 01:19:11.422387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.422341 4552 generic.go:356] "Generic (PLEG): container finished" podID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerID="5736239ae1067d4d4c00a83d4185a4fb2c863c94cac53de40caecce73208d51d" exitCode=0 Nov 02 01:19:11.422842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.422482 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" event={"ID":"af1bbcb8-4880-4150-9c9e-610a1b1535c0","Type":"ContainerDied","Data":"5736239ae1067d4d4c00a83d4185a4fb2c863c94cac53de40caecce73208d51d"} Nov 02 01:19:11.427553 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.427521 4552 generic.go:356] "Generic (PLEG): container finished" podID="826866da-0827-4c60-b767-895198c5e9da" containerID="d1ec84481487cf864858de47b22697725c9567b7f371811a05b3cee84756aada" exitCode=0 Nov 02 01:19:11.427667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.427647 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" event={"ID":"826866da-0827-4c60-b767-895198c5e9da","Type":"ContainerDied","Data":"d1ec84481487cf864858de47b22697725c9567b7f371811a05b3cee84756aada"} Nov 02 01:19:11.432071 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.432036 4552 generic.go:356] "Generic (PLEG): container finished" podID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerID="f1a33245a064e7a3ebd638de88c6b416b2bb4942c1d10c7a23a490a8cef58b45" exitCode=0 Nov 02 01:19:11.432181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.432153 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" event={"ID":"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3","Type":"ContainerDied","Data":"f1a33245a064e7a3ebd638de88c6b416b2bb4942c1d10c7a23a490a8cef58b45"} Nov 02 01:19:11.435074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.435048 4552 generic.go:356] "Generic (PLEG): container finished" podID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerID="05dd7dea02dbdcd0a0f472a4d26c0f2f720787f11cbf70dc5cfd98426d285fdc" exitCode=0 Nov 02 01:19:11.436202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:11.436143 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" event={"ID":"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44","Type":"ContainerDied","Data":"05dd7dea02dbdcd0a0f472a4d26c0f2f720787f11cbf70dc5cfd98426d285fdc"} Nov 02 01:19:12.446206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:12.445402 4552 generic.go:356] "Generic (PLEG): container finished" podID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerID="3d4560a9562f0d3d03c599c4d85bbf773769435436252dec90c6ae3b8fac39be" exitCode=0 Nov 02 01:19:12.446206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:12.445470 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" event={"ID":"3db39889-40f5-4ad2-9fae-22f28e342c75","Type":"ContainerDied","Data":"3d4560a9562f0d3d03c599c4d85bbf773769435436252dec90c6ae3b8fac39be"} Nov 02 01:19:12.454214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:12.453448 4552 generic.go:356] "Generic (PLEG): container finished" podID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerID="46d8866825ea6edc5164b9ccf6c39c65ae1f7f742d3e9b7b7f6a8f83c771a62f" exitCode=0 Nov 02 01:19:12.454214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:12.453551 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" event={"ID":"d59c6a57-e714-4ba4-a553-bffb240b26bb","Type":"ContainerDied","Data":"46d8866825ea6edc5164b9ccf6c39c65ae1f7f742d3e9b7b7f6a8f83c771a62f"} Nov 02 01:19:12.461621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:12.460757 4552 generic.go:356] "Generic (PLEG): container finished" podID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerID="80441ecf315a61d3181fc1de324356fb1cab9d8df4d76cba95fc993eb735f834" exitCode=0 Nov 02 01:19:12.461621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:12.460863 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" event={"ID":"b868b2f8-9412-459a-8c5d-c28617e5e341","Type":"ContainerDied","Data":"80441ecf315a61d3181fc1de324356fb1cab9d8df4d76cba95fc993eb735f834"} Nov 02 01:19:17.519886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:17.519846 4552 generic.go:356] "Generic (PLEG): container finished" podID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerID="81c30c2ba6b3f9a91b7aa0923e20b3fa4d88eb5edcc978da2b9126b540fa5b4e" exitCode=0 Nov 02 01:19:17.520491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:17.519898 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" event={"ID":"03882b08-3940-4e35-8eb2-2fc44b83cb90","Type":"ContainerDied","Data":"81c30c2ba6b3f9a91b7aa0923e20b3fa4d88eb5edcc978da2b9126b540fa5b4e"} Nov 02 01:19:19.551323 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:19.551277 4552 generic.go:356] "Generic (PLEG): container finished" podID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerID="7a063ad48e9697c5392fc524af93927e83f4851fedcb9651c76aab1c24e9f09b" exitCode=0 Nov 02 01:19:19.551849 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:19.551380 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" event={"ID":"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3","Type":"ContainerDied","Data":"7a063ad48e9697c5392fc524af93927e83f4851fedcb9651c76aab1c24e9f09b"} Nov 02 01:19:27.787363 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:19:27.787322 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:19:30.954149 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:19:30.954063 4552 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d22a26b_d71e_4005_9c4d_2e6a59ab9a4b.slice/crio-a71fb4a0c8b04c651d90fa64e0741329c0010873b967d2b830ea9501784adec2.scope\": RecentStats: unable to find data in memory cache]" Nov 02 01:19:31.718434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:31.718395 4552 generic.go:356] "Generic (PLEG): container finished" podID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerID="a71fb4a0c8b04c651d90fa64e0741329c0010873b967d2b830ea9501784adec2" exitCode=0 Nov 02 01:19:31.718611 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:31.718507 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" event={"ID":"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b","Type":"ContainerDied","Data":"a71fb4a0c8b04c651d90fa64e0741329c0010873b967d2b830ea9501784adec2"} Nov 02 01:19:31.727541 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:31.727514 4552 generic.go:356] "Generic (PLEG): container finished" podID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerID="467a27a60f657d3578f8a8ca517de6bad8eb285758b0b091adc64e99b2e19bac" exitCode=0 Nov 02 01:19:31.727678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:31.727565 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" event={"ID":"0f34e9cf-87c2-426c-8915-55f5a9dd0fea","Type":"ContainerDied","Data":"467a27a60f657d3578f8a8ca517de6bad8eb285758b0b091adc64e99b2e19bac"} Nov 02 01:19:32.735982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.735941 4552 generic.go:356] "Generic (PLEG): container finished" podID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerID="d3248da5f1f91c41bab403605dce845e5fe3ad0c846fe132ebe5be99ca6e57f2" exitCode=0 Nov 02 01:19:32.736508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.736141 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" event={"ID":"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44","Type":"ContainerDied","Data":"d3248da5f1f91c41bab403605dce845e5fe3ad0c846fe132ebe5be99ca6e57f2"} Nov 02 01:19:32.740194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.740161 4552 generic.go:356] "Generic (PLEG): container finished" podID="826866da-0827-4c60-b767-895198c5e9da" containerID="c47f3a477f56bc324691cb14e28e29429fdd882953d91a7752e5e7d494416da3" exitCode=0 Nov 02 01:19:32.740350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.740283 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" event={"ID":"826866da-0827-4c60-b767-895198c5e9da","Type":"ContainerDied","Data":"c47f3a477f56bc324691cb14e28e29429fdd882953d91a7752e5e7d494416da3"} Nov 02 01:19:32.745351 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.745323 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" event={"ID":"d59c6a57-e714-4ba4-a553-bffb240b26bb","Type":"ContainerStarted","Data":"e3c42da60e73d11db55e28ca724c859df85a76578e5e478aa4fe1407b092e849"} Nov 02 01:19:32.749238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.749214 4552 generic.go:356] "Generic (PLEG): container finished" podID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerID="763ca2258b1bf9d8ee58f5815717d35e18cc891f926183b9595bff9996f38d9b" exitCode=0 Nov 02 01:19:32.749610 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.749585 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" event={"ID":"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3","Type":"ContainerDied","Data":"763ca2258b1bf9d8ee58f5815717d35e18cc891f926183b9595bff9996f38d9b"} Nov 02 01:19:32.833133 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.833069 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" podStartSLOduration=3.030369586 podStartE2EDuration="45.833052686s" totalImagesPullingTime="42.8026831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:32.803980109 +0000 UTC m=+272.909769029" watchObservedRunningTime="2026-11-02 01:19:32.833052686 +0000 UTC m=+272.938841601" Nov 02 01:19:32.977737 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:32.977707 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:19:33.019418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.019382 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:19:33.045267 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.045060 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv\" (UniqueName: \"kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv\") pod \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " Nov 02 01:19:33.045521 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.045499 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util\") pod \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " Nov 02 01:19:33.045624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.045607 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle\") pod \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\" (UID: \"0f34e9cf-87c2-426c-8915-55f5a9dd0fea\") " Nov 02 01:19:33.049061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.046233 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle" pod "0f34e9cf-87c2-426c-8915-55f5a9dd0fea" (UID: "0f34e9cf-87c2-426c-8915-55f5a9dd0fea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:33.049187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.049154 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv" pod "0f34e9cf-87c2-426c-8915-55f5a9dd0fea" (UID: "0f34e9cf-87c2-426c-8915-55f5a9dd0fea"). InnerVolumeSpecName "kube-api-access-bnvgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:33.053042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.053014 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util" pod "0f34e9cf-87c2-426c-8915-55f5a9dd0fea" (UID: "0f34e9cf-87c2-426c-8915-55f5a9dd0fea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:33.146936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.146901 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle\") pod \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " Nov 02 01:19:33.147145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.146989 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht\" (UniqueName: \"kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht\") pod \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " Nov 02 01:19:33.147145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.147016 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util\") pod \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\" (UID: \"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b\") " Nov 02 01:19:33.147367 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.147348 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvgv\" (UniqueName: \"kubernetes.io/projected/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-kube-api-access-bnvgv\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:33.147433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.147372 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:33.147433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.147387 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f34e9cf-87c2-426c-8915-55f5a9dd0fea-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:33.147853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.147821 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle" pod "8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" (UID: "8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:33.149691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.149653 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht" pod "8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" (UID: "8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b"). InnerVolumeSpecName "kube-api-access-j8mht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:33.155292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.155266 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util" pod "8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" (UID: "8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:33.248654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.248617 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mht\" (UniqueName: \"kubernetes.io/projected/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-kube-api-access-j8mht\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:33.248654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.248647 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:33.248654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.248657 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:33.756857 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.756833 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" Nov 02 01:19:33.757349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.756864 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfmvs" event={"ID":"8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b","Type":"ContainerDied","Data":"88e2da7b59f51cdb2ae28a372aa059cea482db68ac436f791be0231235414ea2"} Nov 02 01:19:33.757349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.756918 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e2da7b59f51cdb2ae28a372aa059cea482db68ac436f791be0231235414ea2" Nov 02 01:19:33.759673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.759561 4552 generic.go:356] "Generic (PLEG): container finished" podID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerID="a91d5907b596b9e45031bee147e6402dffc62a8e398c155bfd1878fad4db8e40" exitCode=0 Nov 02 01:19:33.759673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.759602 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" event={"ID":"3db39889-40f5-4ad2-9fae-22f28e342c75","Type":"ContainerDied","Data":"a91d5907b596b9e45031bee147e6402dffc62a8e398c155bfd1878fad4db8e40"} Nov 02 01:19:33.762292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.762263 4552 generic.go:356] "Generic (PLEG): container finished" podID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerID="e3c42da60e73d11db55e28ca724c859df85a76578e5e478aa4fe1407b092e849" exitCode=0 Nov 02 01:19:33.762418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.762367 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" event={"ID":"d59c6a57-e714-4ba4-a553-bffb240b26bb","Type":"ContainerDied","Data":"e3c42da60e73d11db55e28ca724c859df85a76578e5e478aa4fe1407b092e849"} Nov 02 01:19:33.764665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.764639 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" event={"ID":"0f34e9cf-87c2-426c-8915-55f5a9dd0fea","Type":"ContainerDied","Data":"f854e31655847ef0b42a4c9c02cf4fc01560d1af5170ebf296dae3275da2ebdf"} Nov 02 01:19:33.764788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.764686 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f854e31655847ef0b42a4c9c02cf4fc01560d1af5170ebf296dae3275da2ebdf" Nov 02 01:19:33.764788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.764651 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ssth6" Nov 02 01:19:33.767630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.767604 4552 generic.go:356] "Generic (PLEG): container finished" podID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerID="a0155bdd067dc4409514b581b6eda08b85317c707625e00ca95a8bb27215d99e" exitCode=0 Nov 02 01:19:33.767745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.767672 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" event={"ID":"af1bbcb8-4880-4150-9c9e-610a1b1535c0","Type":"ContainerDied","Data":"a0155bdd067dc4409514b581b6eda08b85317c707625e00ca95a8bb27215d99e"} Nov 02 01:19:33.954317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.954287 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:19:33.982510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.982484 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:19:33.984719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:33.984697 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:19:34.056638 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.056610 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle\") pod \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " Nov 02 01:19:34.056741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.056657 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util\") pod \"826866da-0827-4c60-b767-895198c5e9da\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " Nov 02 01:19:34.056741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.056690 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle\") pod \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " Nov 02 01:19:34.056741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.056722 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w\" (UniqueName: \"kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w\") pod \"826866da-0827-4c60-b767-895198c5e9da\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " Nov 02 01:19:34.056922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.056792 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7\" (UniqueName: \"kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7\") pod \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " Nov 02 01:19:34.056922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.056824 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util\") pod \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " Nov 02 01:19:34.057065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.057030 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle" pod "20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" (UID: "20a0716b-f78f-41b9-b18d-e2c97cbe8ba3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:34.057202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.057083 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util\") pod \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\" (UID: \"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3\") " Nov 02 01:19:34.057202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.057161 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh\" (UniqueName: \"kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh\") pod \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\" (UID: \"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44\") " Nov 02 01:19:34.057315 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.057234 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle\") pod \"826866da-0827-4c60-b767-895198c5e9da\" (UID: \"826866da-0827-4c60-b767-895198c5e9da\") " Nov 02 01:19:34.057315 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.057243 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle" pod "8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" (UID: "8f5d1e6b-22a6-43d5-a54a-17f4d6848d44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:34.059167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.058224 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.059167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.058249 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.059167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.058707 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle" pod "826866da-0827-4c60-b767-895198c5e9da" (UID: "826866da-0827-4c60-b767-895198c5e9da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:34.060072 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.060028 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7" pod "20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" (UID: "20a0716b-f78f-41b9-b18d-e2c97cbe8ba3"). InnerVolumeSpecName "kube-api-access-b2xt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:34.060196 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.060139 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w" pod "826866da-0827-4c60-b767-895198c5e9da" (UID: "826866da-0827-4c60-b767-895198c5e9da"). InnerVolumeSpecName "kube-api-access-r5s5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:34.061403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.061377 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh" pod "8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" (UID: "8f5d1e6b-22a6-43d5-a54a-17f4d6848d44"). InnerVolumeSpecName "kube-api-access-xfplh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:34.065571 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.065544 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util" pod "826866da-0827-4c60-b767-895198c5e9da" (UID: "826866da-0827-4c60-b767-895198c5e9da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:34.065991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.065964 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util" pod "8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" (UID: "8f5d1e6b-22a6-43d5-a54a-17f4d6848d44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:34.067000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.066974 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util" pod "20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" (UID: "20a0716b-f78f-41b9-b18d-e2c97cbe8ba3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:34.159585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159559 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.159717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159590 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5s5w\" (UniqueName: \"kubernetes.io/projected/826866da-0827-4c60-b767-895198c5e9da-kube-api-access-r5s5w\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.159717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159608 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xt7\" (UniqueName: \"kubernetes.io/projected/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-kube-api-access-b2xt7\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.159717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159622 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.159717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159637 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20a0716b-f78f-41b9-b18d-e2c97cbe8ba3-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.159717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159647 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfplh\" (UniqueName: \"kubernetes.io/projected/8f5d1e6b-22a6-43d5-a54a-17f4d6848d44-kube-api-access-xfplh\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.159717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.159655 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826866da-0827-4c60-b767-895198c5e9da-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:34.774563 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.774536 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" event={"ID":"826866da-0827-4c60-b767-895198c5e9da","Type":"ContainerDied","Data":"d74418e4be15b9bfcd4da3a6fef44356a02b2238036746e382aa8eb1a7619126"} Nov 02 01:19:34.774887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.774583 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74418e4be15b9bfcd4da3a6fef44356a02b2238036746e382aa8eb1a7619126" Nov 02 01:19:34.774887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.774546 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7145zvq" Nov 02 01:19:34.776813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.776788 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" event={"ID":"03882b08-3940-4e35-8eb2-2fc44b83cb90","Type":"ContainerStarted","Data":"9763deb10752c152316bf12fc313aaa4c66000ef7539627c0cd3e5ce4237095b"} Nov 02 01:19:34.778634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.778618 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" Nov 02 01:19:34.778746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.778650 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446tgrg" event={"ID":"20a0716b-f78f-41b9-b18d-e2c97cbe8ba3","Type":"ContainerDied","Data":"827c31738fc5c3535d6cd7a776cb6d848a652a6272acba18fe0445cf0edf2a46"} Nov 02 01:19:34.778746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.778692 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827c31738fc5c3535d6cd7a776cb6d848a652a6272acba18fe0445cf0edf2a46" Nov 02 01:19:34.780766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.780747 4552 generic.go:356] "Generic (PLEG): container finished" podID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerID="5bb00845b258a439fb713b3c7c137490a93b009a046a4c8412af986df3d18a09" exitCode=0 Nov 02 01:19:34.780871 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.780809 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" event={"ID":"b868b2f8-9412-459a-8c5d-c28617e5e341","Type":"ContainerDied","Data":"5bb00845b258a439fb713b3c7c137490a93b009a046a4c8412af986df3d18a09"} Nov 02 01:19:34.782831 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.782814 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" Nov 02 01:19:34.782917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.782849 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6689gg" event={"ID":"8f5d1e6b-22a6-43d5-a54a-17f4d6848d44","Type":"ContainerDied","Data":"436ecc34fa4fec81aebd67ec83dfd94663d78761ca3602a15094e0d307094470"} Nov 02 01:19:34.782917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.782882 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="436ecc34fa4fec81aebd67ec83dfd94663d78761ca3602a15094e0d307094470" Nov 02 01:19:34.922605 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.922581 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:19:34.973624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.973602 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:19:34.976495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:34.976479 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:19:35.067251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067218 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg\" (UniqueName: \"kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg\") pod \"3db39889-40f5-4ad2-9fae-22f28e342c75\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " Nov 02 01:19:35.067251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067254 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle\") pod \"3db39889-40f5-4ad2-9fae-22f28e342c75\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " Nov 02 01:19:35.067487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067280 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util\") pod \"3db39889-40f5-4ad2-9fae-22f28e342c75\" (UID: \"3db39889-40f5-4ad2-9fae-22f28e342c75\") " Nov 02 01:19:35.067487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067302 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util\") pod \"d59c6a57-e714-4ba4-a553-bffb240b26bb\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " Nov 02 01:19:35.067487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067341 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb\" (UniqueName: \"kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb\") pod \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " Nov 02 01:19:35.067487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067358 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util\") pod \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " Nov 02 01:19:35.067487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067410 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle\") pod \"d59c6a57-e714-4ba4-a553-bffb240b26bb\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " Nov 02 01:19:35.067487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067483 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72\" (UniqueName: \"kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72\") pod \"d59c6a57-e714-4ba4-a553-bffb240b26bb\" (UID: \"d59c6a57-e714-4ba4-a553-bffb240b26bb\") " Nov 02 01:19:35.067804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067513 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle\") pod \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\" (UID: \"af1bbcb8-4880-4150-9c9e-610a1b1535c0\") " Nov 02 01:19:35.067889 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.067857 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle" pod "3db39889-40f5-4ad2-9fae-22f28e342c75" (UID: "3db39889-40f5-4ad2-9fae-22f28e342c75"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:35.068604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.068578 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle" pod "af1bbcb8-4880-4150-9c9e-610a1b1535c0" (UID: "af1bbcb8-4880-4150-9c9e-610a1b1535c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:35.068783 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.068751 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle" pod "d59c6a57-e714-4ba4-a553-bffb240b26bb" (UID: "d59c6a57-e714-4ba4-a553-bffb240b26bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:35.070405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.070376 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg" pod "3db39889-40f5-4ad2-9fae-22f28e342c75" (UID: "3db39889-40f5-4ad2-9fae-22f28e342c75"). InnerVolumeSpecName "kube-api-access-7qmmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:35.070504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.070377 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb" pod "af1bbcb8-4880-4150-9c9e-610a1b1535c0" (UID: "af1bbcb8-4880-4150-9c9e-610a1b1535c0"). InnerVolumeSpecName "kube-api-access-sscbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:35.070825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.070801 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72" pod "d59c6a57-e714-4ba4-a553-bffb240b26bb" (UID: "d59c6a57-e714-4ba4-a553-bffb240b26bb"). InnerVolumeSpecName "kube-api-access-pkx72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:35.074486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.074464 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util" pod "d59c6a57-e714-4ba4-a553-bffb240b26bb" (UID: "d59c6a57-e714-4ba4-a553-bffb240b26bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:35.074829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.074806 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util" pod "af1bbcb8-4880-4150-9c9e-610a1b1535c0" (UID: "af1bbcb8-4880-4150-9c9e-610a1b1535c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:35.074894 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.074810 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util" pod "3db39889-40f5-4ad2-9fae-22f28e342c75" (UID: "3db39889-40f5-4ad2-9fae-22f28e342c75"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168430 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qmmg\" (UniqueName: \"kubernetes.io/projected/3db39889-40f5-4ad2-9fae-22f28e342c75-kube-api-access-7qmmg\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168451 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168461 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3db39889-40f5-4ad2-9fae-22f28e342c75-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168469 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168477 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscbb\" (UniqueName: \"kubernetes.io/projected/af1bbcb8-4880-4150-9c9e-610a1b1535c0-kube-api-access-sscbb\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168485 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168494 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59c6a57-e714-4ba4-a553-bffb240b26bb-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168502 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkx72\" (UniqueName: \"kubernetes.io/projected/d59c6a57-e714-4ba4-a553-bffb240b26bb-kube-api-access-pkx72\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.168752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.168512 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1bbcb8-4880-4150-9c9e-610a1b1535c0-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:35.788875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.788838 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" event={"ID":"3db39889-40f5-4ad2-9fae-22f28e342c75","Type":"ContainerDied","Data":"cb31ba228fbd59e9d758997b86263678623f5e31c84df6a3fcda2886731d8db7"} Nov 02 01:19:35.789327 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.788883 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb31ba228fbd59e9d758997b86263678623f5e31c84df6a3fcda2886731d8db7" Nov 02 01:19:35.789327 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.788850 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vnvzb" Nov 02 01:19:35.790846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.790825 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" Nov 02 01:19:35.790846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.790834 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx2tn" event={"ID":"d59c6a57-e714-4ba4-a553-bffb240b26bb","Type":"ContainerDied","Data":"0a67e36716ccc356ddf7eb7d123ea768e4f1301aebd510145d09db93fca9b77c"} Nov 02 01:19:35.791006 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.790877 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a67e36716ccc356ddf7eb7d123ea768e4f1301aebd510145d09db93fca9b77c" Nov 02 01:19:35.792891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.792874 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" Nov 02 01:19:35.793007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.792915 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a92fhf" event={"ID":"af1bbcb8-4880-4150-9c9e-610a1b1535c0","Type":"ContainerDied","Data":"3058eeed11b6675f6b644b09768522df981803328acefe2617c5b5eed4d9a868"} Nov 02 01:19:35.793007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.792957 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3058eeed11b6675f6b644b09768522df981803328acefe2617c5b5eed4d9a868" Nov 02 01:19:35.795322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.795298 4552 generic.go:356] "Generic (PLEG): container finished" podID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerID="9763deb10752c152316bf12fc313aaa4c66000ef7539627c0cd3e5ce4237095b" exitCode=0 Nov 02 01:19:35.795427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.795366 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" event={"ID":"03882b08-3940-4e35-8eb2-2fc44b83cb90","Type":"ContainerDied","Data":"9763deb10752c152316bf12fc313aaa4c66000ef7539627c0cd3e5ce4237095b"} Nov 02 01:19:35.922187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.922164 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:19:35.951451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:35.951429 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:19:36.075244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075148 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle\") pod \"b868b2f8-9412-459a-8c5d-c28617e5e341\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " Nov 02 01:19:36.075244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075199 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util\") pod \"b868b2f8-9412-459a-8c5d-c28617e5e341\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " Nov 02 01:19:36.075244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075241 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm\" (UniqueName: \"kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm\") pod \"03882b08-3940-4e35-8eb2-2fc44b83cb90\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " Nov 02 01:19:36.075540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075273 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765\" (UniqueName: \"kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765\") pod \"b868b2f8-9412-459a-8c5d-c28617e5e341\" (UID: \"b868b2f8-9412-459a-8c5d-c28617e5e341\") " Nov 02 01:19:36.075540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075313 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle\") pod \"03882b08-3940-4e35-8eb2-2fc44b83cb90\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " Nov 02 01:19:36.075540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075330 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util\") pod \"03882b08-3940-4e35-8eb2-2fc44b83cb90\" (UID: \"03882b08-3940-4e35-8eb2-2fc44b83cb90\") " Nov 02 01:19:36.075862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.075832 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle" pod "b868b2f8-9412-459a-8c5d-c28617e5e341" (UID: "b868b2f8-9412-459a-8c5d-c28617e5e341"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:36.076333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.076304 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle" pod "03882b08-3940-4e35-8eb2-2fc44b83cb90" (UID: "03882b08-3940-4e35-8eb2-2fc44b83cb90"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:36.077642 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.077616 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm" pod "03882b08-3940-4e35-8eb2-2fc44b83cb90" (UID: "03882b08-3940-4e35-8eb2-2fc44b83cb90"). InnerVolumeSpecName "kube-api-access-nlktm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:36.077929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.077901 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765" pod "b868b2f8-9412-459a-8c5d-c28617e5e341" (UID: "b868b2f8-9412-459a-8c5d-c28617e5e341"). InnerVolumeSpecName "kube-api-access-rc765". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:36.081474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.081454 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util" pod "b868b2f8-9412-459a-8c5d-c28617e5e341" (UID: "b868b2f8-9412-459a-8c5d-c28617e5e341"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:36.081896 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.081875 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util" pod "03882b08-3940-4e35-8eb2-2fc44b83cb90" (UID: "03882b08-3940-4e35-8eb2-2fc44b83cb90"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:36.176789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.176748 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:36.176789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.176784 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b868b2f8-9412-459a-8c5d-c28617e5e341-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:36.176789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.176798 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlktm\" (UniqueName: \"kubernetes.io/projected/03882b08-3940-4e35-8eb2-2fc44b83cb90-kube-api-access-nlktm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:36.176983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.176811 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc765\" (UniqueName: \"kubernetes.io/projected/b868b2f8-9412-459a-8c5d-c28617e5e341-kube-api-access-rc765\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:36.176983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.176823 4552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-bundle\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:36.176983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.176834 4552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03882b08-3940-4e35-8eb2-2fc44b83cb90-util\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:36.802298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.802266 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" Nov 02 01:19:36.802298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.802289 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srxh4" event={"ID":"b868b2f8-9412-459a-8c5d-c28617e5e341","Type":"ContainerDied","Data":"e4484b20e48a4d3cb9a936460600ff088968f4a12c07732f6d27e3300fa5a226"} Nov 02 01:19:36.802828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.802333 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4484b20e48a4d3cb9a936460600ff088968f4a12c07732f6d27e3300fa5a226" Nov 02 01:19:36.804284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.804260 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" event={"ID":"03882b08-3940-4e35-8eb2-2fc44b83cb90","Type":"ContainerDied","Data":"824aef69930aae783e84b5a10b2bb83ba2df9afd78c34ec336d61ce83008559f"} Nov 02 01:19:36.804409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.804294 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="824aef69930aae783e84b5a10b2bb83ba2df9afd78c34ec336d61ce83008559f" Nov 02 01:19:36.804409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:36.804303 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctllz5" Nov 02 01:19:38.489846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.489812 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7"] Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490150 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="util" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490162 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="util" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490173 4552 cpu_manager.go:414] "containerMap: removing container" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="util" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490177 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="util" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490184 4552 cpu_manager.go:414] "containerMap: removing container" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="extract" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490189 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="extract" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490196 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="extract" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490200 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="extract" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490207 4552 cpu_manager.go:414] "containerMap: removing container" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="extract" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490211 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="extract" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490219 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="pull" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490222 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="pull" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490228 4552 cpu_manager.go:414] "containerMap: removing container" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="pull" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490232 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="pull" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490240 4552 cpu_manager.go:414] "containerMap: removing container" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="util" Nov 02 01:19:38.490241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490244 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490251 4552 cpu_manager.go:414] "containerMap: removing container" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490257 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490263 4552 cpu_manager.go:414] "containerMap: removing container" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490267 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490276 4552 cpu_manager.go:414] "containerMap: removing container" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490280 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490286 4552 cpu_manager.go:414] "containerMap: removing container" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490290 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490297 4552 cpu_manager.go:414] "containerMap: removing container" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490301 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490306 4552 cpu_manager.go:414] "containerMap: removing container" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490311 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490316 4552 cpu_manager.go:414] "containerMap: removing container" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490321 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490326 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490330 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490338 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490342 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490349 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490354 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490359 4552 cpu_manager.go:414] "containerMap: removing container" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490363 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490370 4552 cpu_manager.go:414] "containerMap: removing container" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490374 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490379 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490383 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490390 4552 cpu_manager.go:414] "containerMap: removing container" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490395 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490401 4552 cpu_manager.go:414] "containerMap: removing container" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490405 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490415 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490419 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490425 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490429 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490434 4552 cpu_manager.go:414] "containerMap: removing container" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490439 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490444 4552 cpu_manager.go:414] "containerMap: removing container" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490448 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490455 4552 cpu_manager.go:414] "containerMap: removing container" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490459 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="util" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490464 4552 cpu_manager.go:414] "containerMap: removing container" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490468 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="pull" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490473 4552 cpu_manager.go:414] "containerMap: removing container" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490477 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490519 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d22a26b-d71e-4005-9c4d-2e6a59ab9a4b" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490526 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59c6a57-e714-4ba4-a553-bffb240b26bb" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490532 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a0716b-f78f-41b9-b18d-e2c97cbe8ba3" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490538 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1bbcb8-4880-4150-9c9e-610a1b1535c0" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490544 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="b868b2f8-9412-459a-8c5d-c28617e5e341" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490550 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f34e9cf-87c2-426c-8915-55f5a9dd0fea" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490555 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5d1e6b-22a6-43d5-a54a-17f4d6848d44" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490560 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db39889-40f5-4ad2-9fae-22f28e342c75" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490566 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="826866da-0827-4c60-b767-895198c5e9da" containerName="extract" Nov 02 01:19:38.490706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.490571 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="03882b08-3940-4e35-8eb2-2fc44b83cb90" containerName="extract" Nov 02 01:19:38.493092 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.493077 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.499265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.499239 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 01:19:38.499265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.499253 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 01:19:38.499511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.499473 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 01:19:38.499511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.499487 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vj7qc\"" Nov 02 01:19:38.500148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.499564 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 01:19:38.500148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.499569 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 01:19:38.528775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.528751 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7"] Nov 02 01:19:38.598228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.598200 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60baacdd-b56d-4672-bc18-e3caf6ab4be4-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.598366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.598235 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.598366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.598262 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.598366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.598321 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfrl\" (UniqueName: \"kubernetes.io/projected/60baacdd-b56d-4672-bc18-e3caf6ab4be4-kube-api-access-jnfrl\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.598501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.598480 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.598533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.598520 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60baacdd-b56d-4672-bc18-e3caf6ab4be4-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.699828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.699796 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.699961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.699837 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60baacdd-b56d-4672-bc18-e3caf6ab4be4-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.700006 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.699955 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60baacdd-b56d-4672-bc18-e3caf6ab4be4-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.700006 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.699997 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.700069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.700042 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.700107 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.700067 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfrl\" (UniqueName: \"kubernetes.io/projected/60baacdd-b56d-4672-bc18-e3caf6ab4be4-kube-api-access-jnfrl\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.700374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.700349 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60baacdd-b56d-4672-bc18-e3caf6ab4be4-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.702467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.702440 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60baacdd-b56d-4672-bc18-e3caf6ab4be4-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.702546 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.702476 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.702698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.702675 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.702730 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.702717 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60baacdd-b56d-4672-bc18-e3caf6ab4be4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.716186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.716164 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfrl\" (UniqueName: \"kubernetes.io/projected/60baacdd-b56d-4672-bc18-e3caf6ab4be4-kube-api-access-jnfrl\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7\" (UID: \"60baacdd-b56d-4672-bc18-e3caf6ab4be4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:38.805976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:38.805912 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:39.106361 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:39.106337 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7"] Nov 02 01:19:39.108094 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:19:39.108068 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60baacdd_b56d_4672_bc18_e3caf6ab4be4.slice/crio-8828c11b229fe0979e39627676e2d1499f5333ebfdcdae2e728fbd64a1a279d0 WatchSource:0}: Error finding container 8828c11b229fe0979e39627676e2d1499f5333ebfdcdae2e728fbd64a1a279d0: Status 404 returned error can't find the container with id 8828c11b229fe0979e39627676e2d1499f5333ebfdcdae2e728fbd64a1a279d0 Nov 02 01:19:39.822523 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:39.822484 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" event={"ID":"60baacdd-b56d-4672-bc18-e3caf6ab4be4","Type":"ContainerStarted","Data":"8828c11b229fe0979e39627676e2d1499f5333ebfdcdae2e728fbd64a1a279d0"} Nov 02 01:19:43.845583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:43.845544 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" event={"ID":"60baacdd-b56d-4672-bc18-e3caf6ab4be4","Type":"ContainerStarted","Data":"0885ac2e3179664916dbd83e90d53a1a80cca2a0a0dcd1c34da0243bd5f88732"} Nov 02 01:19:44.145434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.145362 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2"] Nov 02 01:19:44.147955 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.147935 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.150375 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.150352 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e05de3d-7173-4ca5-ab1e-55872fbe34db-tmp\") pod \"jobset-operator-674cd59dc5-44bc2\" (UID: \"9e05de3d-7173-4ca5-ab1e-55872fbe34db\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.150481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.150422 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmd5\" (UniqueName: \"kubernetes.io/projected/9e05de3d-7173-4ca5-ab1e-55872fbe34db-kube-api-access-8cmd5\") pod \"jobset-operator-674cd59dc5-44bc2\" (UID: \"9e05de3d-7173-4ca5-ab1e-55872fbe34db\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.151742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.151722 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:19:44.151817 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.151733 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 01:19:44.158788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.158767 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2"] Nov 02 01:19:44.188667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.188636 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-m7s9b\"" Nov 02 01:19:44.251154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.251127 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmd5\" (UniqueName: \"kubernetes.io/projected/9e05de3d-7173-4ca5-ab1e-55872fbe34db-kube-api-access-8cmd5\") pod \"jobset-operator-674cd59dc5-44bc2\" (UID: \"9e05de3d-7173-4ca5-ab1e-55872fbe34db\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.251291 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.251162 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e05de3d-7173-4ca5-ab1e-55872fbe34db-tmp\") pod \"jobset-operator-674cd59dc5-44bc2\" (UID: \"9e05de3d-7173-4ca5-ab1e-55872fbe34db\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.251482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.251467 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e05de3d-7173-4ca5-ab1e-55872fbe34db-tmp\") pod \"jobset-operator-674cd59dc5-44bc2\" (UID: \"9e05de3d-7173-4ca5-ab1e-55872fbe34db\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.263740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.263712 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmd5\" (UniqueName: \"kubernetes.io/projected/9e05de3d-7173-4ca5-ab1e-55872fbe34db-kube-api-access-8cmd5\") pod \"jobset-operator-674cd59dc5-44bc2\" (UID: \"9e05de3d-7173-4ca5-ab1e-55872fbe34db\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.468324 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.468292 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" Nov 02 01:19:44.742746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.742724 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2"] Nov 02 01:19:44.744812 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:19:44.744781 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e05de3d_7173_4ca5_ab1e_55872fbe34db.slice/crio-6e7c164c584ea45eeb8a7648acb8eef86d4acfcf94807ad45e269f2a6018fb6b WatchSource:0}: Error finding container 6e7c164c584ea45eeb8a7648acb8eef86d4acfcf94807ad45e269f2a6018fb6b: Status 404 returned error can't find the container with id 6e7c164c584ea45eeb8a7648acb8eef86d4acfcf94807ad45e269f2a6018fb6b Nov 02 01:19:44.851950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.851904 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" event={"ID":"9e05de3d-7173-4ca5-ab1e-55872fbe34db","Type":"ContainerStarted","Data":"6e7c164c584ea45eeb8a7648acb8eef86d4acfcf94807ad45e269f2a6018fb6b"} Nov 02 01:19:44.853611 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.853590 4552 generic.go:356] "Generic (PLEG): container finished" podID="60baacdd-b56d-4672-bc18-e3caf6ab4be4" containerID="0885ac2e3179664916dbd83e90d53a1a80cca2a0a0dcd1c34da0243bd5f88732" exitCode=0 Nov 02 01:19:44.853699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:44.853672 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" event={"ID":"60baacdd-b56d-4672-bc18-e3caf6ab4be4","Type":"ContainerDied","Data":"0885ac2e3179664916dbd83e90d53a1a80cca2a0a0dcd1c34da0243bd5f88732"} Nov 02 01:19:45.860757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:45.860707 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" event={"ID":"60baacdd-b56d-4672-bc18-e3caf6ab4be4","Type":"ContainerStarted","Data":"2dbd30a0b1171118b577b574491d3a46a10cb1973784401d788cc86acc026ba6"} Nov 02 01:19:45.861189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:45.860810 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:45.906416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:45.906369 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" podStartSLOduration=2.819272874 podStartE2EDuration="7.906354688s" totalImagesPullingTime="4.599692423s" totalInitContainerRuntime="487.389391ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:45.904097802 +0000 UTC m=+286.009886714" watchObservedRunningTime="2026-11-02 01:19:45.906354688 +0000 UTC m=+286.012143602" Nov 02 01:19:49.882410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:49.882373 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" event={"ID":"9e05de3d-7173-4ca5-ab1e-55872fbe34db","Type":"ContainerStarted","Data":"113bc90e6fb9d2715ced92789d089e7248281f9f2f86091b20e434684a0b681b"} Nov 02 01:19:49.912588 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:49.912537 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-44bc2" podStartSLOduration=1.835322165 podStartE2EDuration="5.912522973s" totalImagesPullingTime="4.077200808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:49.910573798 +0000 UTC m=+290.016362716" watchObservedRunningTime="2026-11-02 01:19:49.912522973 +0000 UTC m=+290.018311890" Nov 02 01:19:53.424138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.424086 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs"] Nov 02 01:19:53.427238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.427222 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.433872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.433848 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:19:53.433872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.433847 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 01:19:53.434019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.433856 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gq9z7\"" Nov 02 01:19:53.440652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.440632 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs"] Nov 02 01:19:53.526105 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.526073 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fctt\" (UniqueName: \"kubernetes.io/projected/222afa27-cd14-459a-9582-a97fc6e139f1-kube-api-access-7fctt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs\" (UID: \"222afa27-cd14-459a-9582-a97fc6e139f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.526268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.526133 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/222afa27-cd14-459a-9582-a97fc6e139f1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs\" (UID: \"222afa27-cd14-459a-9582-a97fc6e139f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.627102 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.627071 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fctt\" (UniqueName: \"kubernetes.io/projected/222afa27-cd14-459a-9582-a97fc6e139f1-kube-api-access-7fctt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs\" (UID: \"222afa27-cd14-459a-9582-a97fc6e139f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.627239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.627110 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/222afa27-cd14-459a-9582-a97fc6e139f1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs\" (UID: \"222afa27-cd14-459a-9582-a97fc6e139f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.627478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.627463 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/222afa27-cd14-459a-9582-a97fc6e139f1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs\" (UID: \"222afa27-cd14-459a-9582-a97fc6e139f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.641129 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.641095 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fctt\" (UniqueName: \"kubernetes.io/projected/222afa27-cd14-459a-9582-a97fc6e139f1-kube-api-access-7fctt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs\" (UID: \"222afa27-cd14-459a-9582-a97fc6e139f1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:53.739203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:53.739176 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" Nov 02 01:19:54.031668 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:19:54.031633 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222afa27_cd14_459a_9582_a97fc6e139f1.slice/crio-f1c24f7a3dfd96b350f445201ed8a571c672fff197e66bd8fdcc61327b81e134 WatchSource:0}: Error finding container f1c24f7a3dfd96b350f445201ed8a571c672fff197e66bd8fdcc61327b81e134: Status 404 returned error can't find the container with id f1c24f7a3dfd96b350f445201ed8a571c672fff197e66bd8fdcc61327b81e134 Nov 02 01:19:54.041972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:54.041950 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs"] Nov 02 01:19:54.908317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:54.908273 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" event={"ID":"222afa27-cd14-459a-9582-a97fc6e139f1","Type":"ContainerStarted","Data":"f1c24f7a3dfd96b350f445201ed8a571c672fff197e66bd8fdcc61327b81e134"} Nov 02 01:19:56.433830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.433789 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf"] Nov 02 01:19:56.459093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.459057 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf"] Nov 02 01:19:56.459270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.459176 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:19:56.466202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.466180 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lvt9x\"" Nov 02 01:19:56.466344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.466202 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 01:19:56.466344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.466236 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 01:19:56.551137 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.551082 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2nm\" (UniqueName: \"kubernetes.io/projected/4253a47b-9095-4961-aa62-e6a4e4fd513e-kube-api-access-rr2nm\") pod \"limitador-operator-controller-manager-5778bb4449-6w6cf\" (UID: \"4253a47b-9095-4961-aa62-e6a4e4fd513e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:19:56.652002 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.651971 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2nm\" (UniqueName: \"kubernetes.io/projected/4253a47b-9095-4961-aa62-e6a4e4fd513e-kube-api-access-rr2nm\") pod \"limitador-operator-controller-manager-5778bb4449-6w6cf\" (UID: \"4253a47b-9095-4961-aa62-e6a4e4fd513e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:19:56.664962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.664931 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2nm\" (UniqueName: \"kubernetes.io/projected/4253a47b-9095-4961-aa62-e6a4e4fd513e-kube-api-access-rr2nm\") pod \"limitador-operator-controller-manager-5778bb4449-6w6cf\" (UID: \"4253a47b-9095-4961-aa62-e6a4e4fd513e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:19:56.791365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.791338 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:19:56.868881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.868848 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7" Nov 02 01:19:56.920784 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.920747 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" event={"ID":"222afa27-cd14-459a-9582-a97fc6e139f1","Type":"ContainerStarted","Data":"1866ebe05a09c4bf2b7a67c8293331f9fa73da88cc9e7630e2a9b5272cc11cf4"} Nov 02 01:19:56.945286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:56.945223 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-q9mbs" podStartSLOduration=1.270387327 podStartE2EDuration="3.94520561s" totalImagesPullingTime="2.674818283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:56.944922078 +0000 UTC m=+297.050710993" watchObservedRunningTime="2026-11-02 01:19:56.94520561 +0000 UTC m=+297.050994525" Nov 02 01:19:57.076560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:57.076535 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf"] Nov 02 01:19:57.078769 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:19:57.078732 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4253a47b_9095_4961_aa62_e6a4e4fd513e.slice/crio-b4dd2041961d397f9964d29e9fb375207edfbd2e3dca0b39c9d6094dda5512a5 WatchSource:0}: Error finding container b4dd2041961d397f9964d29e9fb375207edfbd2e3dca0b39c9d6094dda5512a5: Status 404 returned error can't find the container with id b4dd2041961d397f9964d29e9fb375207edfbd2e3dca0b39c9d6094dda5512a5 Nov 02 01:19:57.753067 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:19:57.753033 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:19:57.941605 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:19:57.941564 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" event={"ID":"4253a47b-9095-4961-aa62-e6a4e4fd513e","Type":"ContainerStarted","Data":"b4dd2041961d397f9964d29e9fb375207edfbd2e3dca0b39c9d6094dda5512a5"} Nov 02 01:20:00.497162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.497130 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg"] Nov 02 01:20:00.500479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.500460 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:00.515434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.515412 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gj9nz\"" Nov 02 01:20:00.516177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.516156 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg"] Nov 02 01:20:00.588073 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.588032 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826kq\" (UniqueName: \"kubernetes.io/projected/67d4d1aa-9743-4739-b659-94b11f451e2e-kube-api-access-826kq\") pod \"authorino-operator-7b7bb4bc77-lbxvg\" (UID: \"67d4d1aa-9743-4739-b659-94b11f451e2e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:00.689228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.689195 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826kq\" (UniqueName: \"kubernetes.io/projected/67d4d1aa-9743-4739-b659-94b11f451e2e-kube-api-access-826kq\") pod \"authorino-operator-7b7bb4bc77-lbxvg\" (UID: \"67d4d1aa-9743-4739-b659-94b11f451e2e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:00.709549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.709520 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-826kq\" (UniqueName: \"kubernetes.io/projected/67d4d1aa-9743-4739-b659-94b11f451e2e-kube-api-access-826kq\") pod \"authorino-operator-7b7bb4bc77-lbxvg\" (UID: \"67d4d1aa-9743-4739-b659-94b11f451e2e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:00.812615 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:00.812518 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:01.096316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:01.096296 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg"] Nov 02 01:20:01.098074 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:01.098045 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d4d1aa_9743_4739_b659_94b11f451e2e.slice/crio-60c31cc74c3f1aa885dd38169c0ee47321de7f5d64bc827da338be5d9e00d300 WatchSource:0}: Error finding container 60c31cc74c3f1aa885dd38169c0ee47321de7f5d64bc827da338be5d9e00d300: Status 404 returned error can't find the container with id 60c31cc74c3f1aa885dd38169c0ee47321de7f5d64bc827da338be5d9e00d300 Nov 02 01:20:01.962169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:01.962109 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" event={"ID":"67d4d1aa-9743-4739-b659-94b11f451e2e","Type":"ContainerStarted","Data":"60c31cc74c3f1aa885dd38169c0ee47321de7f5d64bc827da338be5d9e00d300"} Nov 02 01:20:04.043582 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.043555 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf"] Nov 02 01:20:04.046194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.046170 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:04.056087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.056060 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rb2nk\"" Nov 02 01:20:04.056912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.056892 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 01:20:04.066640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.066618 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf"] Nov 02 01:20:04.122957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.122919 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvst\" (UniqueName: \"kubernetes.io/projected/de8461ad-fdf0-462c-ad1f-e1dc0dd490be-kube-api-access-sfvst\") pod \"dns-operator-controller-manager-8557cbdb84-dsrsf\" (UID: \"de8461ad-fdf0-462c-ad1f-e1dc0dd490be\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:04.223625 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.223591 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvst\" (UniqueName: \"kubernetes.io/projected/de8461ad-fdf0-462c-ad1f-e1dc0dd490be-kube-api-access-sfvst\") pod \"dns-operator-controller-manager-8557cbdb84-dsrsf\" (UID: \"de8461ad-fdf0-462c-ad1f-e1dc0dd490be\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:04.238375 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.238346 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvst\" (UniqueName: \"kubernetes.io/projected/de8461ad-fdf0-462c-ad1f-e1dc0dd490be-kube-api-access-sfvst\") pod \"dns-operator-controller-manager-8557cbdb84-dsrsf\" (UID: \"de8461ad-fdf0-462c-ad1f-e1dc0dd490be\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:04.361552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.361474 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:04.642958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.642930 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf"] Nov 02 01:20:04.644463 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:04.644431 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8461ad_fdf0_462c_ad1f_e1dc0dd490be.slice/crio-b6615010c81f95ccbeff788f23003e01537e6ffeeb2b520b40107ef08c7e5f7c WatchSource:0}: Error finding container b6615010c81f95ccbeff788f23003e01537e6ffeeb2b520b40107ef08c7e5f7c: Status 404 returned error can't find the container with id b6615010c81f95ccbeff788f23003e01537e6ffeeb2b520b40107ef08c7e5f7c Nov 02 01:20:04.993763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:04.993728 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" event={"ID":"de8461ad-fdf0-462c-ad1f-e1dc0dd490be","Type":"ContainerStarted","Data":"b6615010c81f95ccbeff788f23003e01537e6ffeeb2b520b40107ef08c7e5f7c"} Nov 02 01:20:06.002648 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:06.002602 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" event={"ID":"67d4d1aa-9743-4739-b659-94b11f451e2e","Type":"ContainerStarted","Data":"0ee7f7293ad2a44faf2c9ec64868056a0e6a5010d960f88889aaa936907c41d0"} Nov 02 01:20:06.003153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:06.002697 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:06.025151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:06.025083 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" podStartSLOduration=1.420060074 podStartE2EDuration="6.025070092s" totalImagesPullingTime="4.605010018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:06.023924906 +0000 UTC m=+306.129713818" watchObservedRunningTime="2026-11-02 01:20:06.025070092 +0000 UTC m=+306.130859006" Nov 02 01:20:09.026750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:09.026709 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" event={"ID":"de8461ad-fdf0-462c-ad1f-e1dc0dd490be","Type":"ContainerStarted","Data":"5da0f12b3e1020e8b4ea89645db421a459969f2364890c360e41ccf93dacbd15"} Nov 02 01:20:09.027255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:09.026814 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:09.052329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:09.052271 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" podStartSLOduration=1.074286429 podStartE2EDuration="5.052250589s" totalImagesPullingTime="3.97796416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:09.045927015 +0000 UTC m=+309.151715941" watchObservedRunningTime="2026-11-02 01:20:09.052250589 +0000 UTC m=+309.158039506" Nov 02 01:20:10.034195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:10.034152 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" event={"ID":"4253a47b-9095-4961-aa62-e6a4e4fd513e","Type":"ContainerStarted","Data":"97b009629c4ec7f8db72b2c18b6ea546ca5db0fc8c31fe42ab503a4959031b51"} Nov 02 01:20:10.034716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:10.034380 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:20:10.056219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:10.056157 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" podStartSLOduration=1.928935182 podStartE2EDuration="14.056141301s" totalImagesPullingTime="12.127206119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:10.054884783 +0000 UTC m=+310.160673696" watchObservedRunningTime="2026-11-02 01:20:10.056141301 +0000 UTC m=+310.161930212" Nov 02 01:20:11.756316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.756280 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg"] Nov 02 01:20:11.759667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.759651 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:11.764500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.764480 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n8fpr\"" Nov 02 01:20:11.771713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.771693 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg"] Nov 02 01:20:11.894013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.893977 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qgwgg\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:11.894219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.894045 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4zl\" (UniqueName: \"kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl\") pod \"kuadrant-operator-controller-manager-678d87766f-qgwgg\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:11.994491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.994461 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4zl\" (UniqueName: \"kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl\") pod \"kuadrant-operator-controller-manager-678d87766f-qgwgg\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:11.994651 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.994522 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qgwgg\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:11.994876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:11.994860 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qgwgg\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:12.010835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:12.010773 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4zl\" (UniqueName: \"kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl\") pod \"kuadrant-operator-controller-manager-678d87766f-qgwgg\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:12.072090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:12.072066 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:12.351396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:12.351375 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg"] Nov 02 01:20:12.353087 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:12.353059 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc610634_845d_4214_96b5_a41d58c809f0.slice/crio-8b6c9fb96a7710999ac9d7919e65034326127b3dbf0b96251fce6cf4ea59497d WatchSource:0}: Error finding container 8b6c9fb96a7710999ac9d7919e65034326127b3dbf0b96251fce6cf4ea59497d: Status 404 returned error can't find the container with id 8b6c9fb96a7710999ac9d7919e65034326127b3dbf0b96251fce6cf4ea59497d Nov 02 01:20:12.355720 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:12.355702 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:20:13.050474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:13.050439 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" event={"ID":"bc610634-845d-4214-96b5-a41d58c809f0","Type":"ContainerStarted","Data":"8b6c9fb96a7710999ac9d7919e65034326127b3dbf0b96251fce6cf4ea59497d"} Nov 02 01:20:16.436935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.436899 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9"] Nov 02 01:20:16.442673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.442650 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.452803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.452025 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zxbxc\"" Nov 02 01:20:16.456750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.456712 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9"] Nov 02 01:20:16.538915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.538863 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.538915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.538915 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.538957 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.538990 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.539022 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fplqq\" (UniqueName: \"kubernetes.io/projected/f1472bbc-66ed-4e4e-830c-87290f40c66a-kube-api-access-fplqq\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.539062 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1472bbc-66ed-4e4e-830c-87290f40c66a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539506 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.539480 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.539541 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.539643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.539620 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.640553 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640515 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.640755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640596 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.640755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640635 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.640755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640663 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.640755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640701 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.640755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640734 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.641022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640781 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fplqq\" (UniqueName: \"kubernetes.io/projected/f1472bbc-66ed-4e4e-830c-87290f40c66a-kube-api-access-fplqq\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.641022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640820 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1472bbc-66ed-4e4e-830c-87290f40c66a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.641022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640891 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.641022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.640919 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.641280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.641264 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.641880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.641854 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.642008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.641960 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1472bbc-66ed-4e4e-830c-87290f40c66a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.642076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.642023 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.644235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.644212 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.651693 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.651665 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.658027 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.657999 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fplqq\" (UniqueName: \"kubernetes.io/projected/f1472bbc-66ed-4e4e-830c-87290f40c66a-kube-api-access-fplqq\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.660503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.660479 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1472bbc-66ed-4e4e-830c-87290f40c66a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b48489g8hq9\" (UID: \"f1472bbc-66ed-4e4e-830c-87290f40c66a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:16.763146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:16.760498 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:17.014244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:17.014168 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lbxvg" Nov 02 01:20:17.074872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:17.074844 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9"] Nov 02 01:20:17.076707 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:17.076675 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1472bbc_66ed_4e4e_830c_87290f40c66a.slice/crio-b3e720fec46e67087fd4de3ca32ea11859e8e095991752bd067b9b001037685e WatchSource:0}: Error finding container b3e720fec46e67087fd4de3ca32ea11859e8e095991752bd067b9b001037685e: Status 404 returned error can't find the container with id b3e720fec46e67087fd4de3ca32ea11859e8e095991752bd067b9b001037685e Nov 02 01:20:17.079492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:17.079443 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:20:17.079574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:17.079544 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:20:17.079611 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:17.079594 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:20:18.080162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.080100 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" event={"ID":"f1472bbc-66ed-4e4e-830c-87290f40c66a","Type":"ContainerStarted","Data":"363d2a9aae0d9f418510b2b998da1ab24b3e4a7a77cac5e3eb265eec89ac36dc"} Nov 02 01:20:18.080641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.080183 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" event={"ID":"f1472bbc-66ed-4e4e-830c-87290f40c66a","Type":"ContainerStarted","Data":"b3e720fec46e67087fd4de3ca32ea11859e8e095991752bd067b9b001037685e"} Nov 02 01:20:18.108469 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.107504 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" podStartSLOduration=2.10748831 podStartE2EDuration="2.10748831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:18.10686667 +0000 UTC m=+318.212655584" watchObservedRunningTime="2026-11-02 01:20:18.10748831 +0000 UTC m=+318.213277225" Nov 02 01:20:18.146766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.146728 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm"] Nov 02 01:20:18.150266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.150241 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.155305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.155283 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 01:20:18.155645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.155481 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 01:20:18.155645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.155616 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lmdzs\"" Nov 02 01:20:18.158230 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.158210 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 01:20:18.162389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.162298 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm"] Nov 02 01:20:18.257481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.257448 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.257657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.257488 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwkl\" (UniqueName: \"kubernetes.io/projected/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-kube-api-access-dqwkl\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.257657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.257520 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.257767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.257660 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.359347 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.359250 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.359347 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.359330 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwkl\" (UniqueName: \"kubernetes.io/projected/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-kube-api-access-dqwkl\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.359572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.359386 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.359572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.359437 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.359847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.359826 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.359917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.359899 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.360290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.360267 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.372154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.372109 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwkl\" (UniqueName: \"kubernetes.io/projected/5e507870-0bc0-4bb7-b0cb-381084e3a5c1-kube-api-access-dqwkl\") pod \"kserve-module-controller-manager-7566cf5fc8-v4xbm\" (UID: \"5e507870-0bc0-4bb7-b0cb-381084e3a5c1\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.465998 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.465961 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:18.756007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.755980 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm"] Nov 02 01:20:18.758201 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:18.758173 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e507870_0bc0_4bb7_b0cb_381084e3a5c1.slice/crio-e80b760b5cb33d53b2144ce2d83d0cd44ee8e8089bcf4408f2761f73c1da5ae3 WatchSource:0}: Error finding container e80b760b5cb33d53b2144ce2d83d0cd44ee8e8089bcf4408f2761f73c1da5ae3: Status 404 returned error can't find the container with id e80b760b5cb33d53b2144ce2d83d0cd44ee8e8089bcf4408f2761f73c1da5ae3 Nov 02 01:20:18.761336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.761307 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:18.767835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:18.767809 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:19.088465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:19.088372 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" event={"ID":"5e507870-0bc0-4bb7-b0cb-381084e3a5c1","Type":"ContainerStarted","Data":"e80b760b5cb33d53b2144ce2d83d0cd44ee8e8089bcf4408f2761f73c1da5ae3"} Nov 02 01:20:19.088909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:19.088677 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:19.089771 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:19.089753 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489g8hq9" Nov 02 01:20:20.045616 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:20.045590 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dsrsf" Nov 02 01:20:21.042627 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:21.042596 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6w6cf" Nov 02 01:20:22.110945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.110908 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" event={"ID":"bc610634-845d-4214-96b5-a41d58c809f0","Type":"ContainerStarted","Data":"7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a"} Nov 02 01:20:22.111438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.111025 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:22.151083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.151033 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" podStartSLOduration=2.348009232 podStartE2EDuration="11.151018769s" totalImagesPullingTime="8.803009537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:22.149165025 +0000 UTC m=+322.254953943" watchObservedRunningTime="2026-11-02 01:20:22.151018769 +0000 UTC m=+322.256807687" Nov 02 01:20:22.382990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.382899 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:20:22.388251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.388222 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.394857 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.394830 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:20:22.399186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.398026 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w92dz\"" Nov 02 01:20:22.399661 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.399352 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 01:20:22.431778 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.431747 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:20:22.505931 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.505892 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5bq\" (UniqueName: \"kubernetes.io/projected/acd9cb01-352d-4902-b532-7f12ae04448f-kube-api-access-cr5bq\") pod \"limitador-limitador-7c9b87d574-8z525\" (UID: \"acd9cb01-352d-4902-b532-7f12ae04448f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.506153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.505961 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd9cb01-352d-4902-b532-7f12ae04448f-config-file\") pod \"limitador-limitador-7c9b87d574-8z525\" (UID: \"acd9cb01-352d-4902-b532-7f12ae04448f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.606639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.606580 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5bq\" (UniqueName: \"kubernetes.io/projected/acd9cb01-352d-4902-b532-7f12ae04448f-kube-api-access-cr5bq\") pod \"limitador-limitador-7c9b87d574-8z525\" (UID: \"acd9cb01-352d-4902-b532-7f12ae04448f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.606814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.606654 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd9cb01-352d-4902-b532-7f12ae04448f-config-file\") pod \"limitador-limitador-7c9b87d574-8z525\" (UID: \"acd9cb01-352d-4902-b532-7f12ae04448f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.607281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.607262 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd9cb01-352d-4902-b532-7f12ae04448f-config-file\") pod \"limitador-limitador-7c9b87d574-8z525\" (UID: \"acd9cb01-352d-4902-b532-7f12ae04448f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.619440 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.619412 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5bq\" (UniqueName: \"kubernetes.io/projected/acd9cb01-352d-4902-b532-7f12ae04448f-kube-api-access-cr5bq\") pod \"limitador-limitador-7c9b87d574-8z525\" (UID: \"acd9cb01-352d-4902-b532-7f12ae04448f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.708398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.708372 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:22.985933 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:22.985906 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:20:22.987893 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:22.987835 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd9cb01_352d_4902_b532_7f12ae04448f.slice/crio-c7cca0e4e24c3152a61ea1df25cedf03ba78f41aa1760b19b39792272ea9413c WatchSource:0}: Error finding container c7cca0e4e24c3152a61ea1df25cedf03ba78f41aa1760b19b39792272ea9413c: Status 404 returned error can't find the container with id c7cca0e4e24c3152a61ea1df25cedf03ba78f41aa1760b19b39792272ea9413c Nov 02 01:20:23.116809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:23.116774 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" event={"ID":"5e507870-0bc0-4bb7-b0cb-381084e3a5c1","Type":"ContainerStarted","Data":"d3dabce86181dc38622844236a8100845fcbf99231fd4597988c56851eb30661"} Nov 02 01:20:23.117334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:23.116970 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:23.118235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:23.118213 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" event={"ID":"acd9cb01-352d-4902-b532-7f12ae04448f","Type":"ContainerStarted","Data":"c7cca0e4e24c3152a61ea1df25cedf03ba78f41aa1760b19b39792272ea9413c"} Nov 02 01:20:23.138373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:23.138325 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" podStartSLOduration=1.283196709 podStartE2EDuration="5.138312387s" totalImagesPullingTime="3.855115678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:23.136963258 +0000 UTC m=+323.242752174" watchObservedRunningTime="2026-11-02 01:20:23.138312387 +0000 UTC m=+323.244101301" Nov 02 01:20:25.135369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:25.135333 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" event={"ID":"acd9cb01-352d-4902-b532-7f12ae04448f","Type":"ContainerStarted","Data":"fd9ca5306cd33e9c17193bc8ab3da0b5cfdaf7b4334e314fba40792ad786e2da"} Nov 02 01:20:25.135764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:25.135420 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:25.155508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:25.155460 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" podStartSLOduration=1.765130182 podStartE2EDuration="3.155447449s" totalImagesPullingTime="1.390317267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:25.152898525 +0000 UTC m=+325.258687436" watchObservedRunningTime="2026-11-02 01:20:25.155447449 +0000 UTC m=+325.261236362" Nov 02 01:20:27.749599 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:20:27.749552 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:20:33.120716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:33.120687 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:20:36.142086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:36.142052 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8z525" Nov 02 01:20:44.128742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:44.128711 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-v4xbm" Nov 02 01:20:57.751399 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:20:57.751358 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:20:58.945370 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.945329 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5ldp5"] Nov 02 01:20:58.950983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.950964 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:58.956684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.956648 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 01:20:58.958303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.958098 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 01:20:58.958303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.958206 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldcrm\"" Nov 02 01:20:58.958303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.958266 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 01:20:58.969041 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:58.969014 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5ldp5"] Nov 02 01:20:59.038364 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.038322 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.038581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.038474 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lzd\" (UniqueName: \"kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.038581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.038512 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.139481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.139437 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.139676 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.139558 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lzd\" (UniqueName: \"kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.139676 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.139596 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.142152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.142098 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.142293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.142226 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.152883 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.152849 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lzd\" (UniqueName: \"kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd\") pod \"maas-controller-579cf7c6d-5ldp5\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.270705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.270670 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:20:59.547157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:20:59.547135 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5ldp5"] Nov 02 01:20:59.549270 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:20:59.549233 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72b5bff_447e_4df5_8519_9e91fda5346f.slice/crio-eb076197386067e3830c8c4387932d1c4bb22abc8bf8a829714ac2ff26042bf4 WatchSource:0}: Error finding container eb076197386067e3830c8c4387932d1c4bb22abc8bf8a829714ac2ff26042bf4: Status 404 returned error can't find the container with id eb076197386067e3830c8c4387932d1c4bb22abc8bf8a829714ac2ff26042bf4 Nov 02 01:21:00.329363 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:00.329325 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" event={"ID":"a72b5bff-447e-4df5-8519-9e91fda5346f","Type":"ContainerStarted","Data":"eb076197386067e3830c8c4387932d1c4bb22abc8bf8a829714ac2ff26042bf4"} Nov 02 01:21:02.426298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:02.426276 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 01:21:03.352212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.352165 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" event={"ID":"a72b5bff-447e-4df5-8519-9e91fda5346f","Type":"ContainerStarted","Data":"db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac"} Nov 02 01:21:03.352453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.352327 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:21:03.374013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.373963 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" podStartSLOduration=2.502022777 podStartE2EDuration="5.373948887s" totalImagesPullingTime="2.87192611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:03.373202373 +0000 UTC m=+363.478991289" watchObservedRunningTime="2026-11-02 01:21:03.373948887 +0000 UTC m=+363.479737800" Nov 02 01:21:03.755545 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.755513 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr"] Nov 02 01:21:03.760614 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.760593 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.769828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.769807 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 01:21:03.770680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.770661 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 01:21:03.774626 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.774606 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 01:21:03.775105 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.775087 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-gqgmw\"" Nov 02 01:21:03.781423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.781362 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr"] Nov 02 01:21:03.889489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.889450 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8ed378a-6a38-423b-a3d2-9c2622a65d42-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.889489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.889486 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5vz\" (UniqueName: \"kubernetes.io/projected/d8ed378a-6a38-423b-a3d2-9c2622a65d42-kube-api-access-lm5vz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.889713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.889517 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed378a-6a38-423b-a3d2-9c2622a65d42-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.889713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.889671 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d8ed378a-6a38-423b-a3d2-9c2622a65d42-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.990331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.990295 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8ed378a-6a38-423b-a3d2-9c2622a65d42-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.990331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.990331 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5vz\" (UniqueName: \"kubernetes.io/projected/d8ed378a-6a38-423b-a3d2-9c2622a65d42-kube-api-access-lm5vz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.990594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.990353 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed378a-6a38-423b-a3d2-9c2622a65d42-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.990594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.990403 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d8ed378a-6a38-423b-a3d2-9c2622a65d42-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.991040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.991015 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed378a-6a38-423b-a3d2-9c2622a65d42-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.991224 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d8ed378a-6a38-423b-a3d2-9c2622a65d42-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:03.992998 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:03.992969 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8ed378a-6a38-423b-a3d2-9c2622a65d42-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:04.002717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:04.002694 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5vz\" (UniqueName: \"kubernetes.io/projected/d8ed378a-6a38-423b-a3d2-9c2622a65d42-kube-api-access-lm5vz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rx7hr\" (UID: \"d8ed378a-6a38-423b-a3d2-9c2622a65d42\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:04.084960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:04.084876 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:04.361817 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:04.361737 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr"] Nov 02 01:21:04.365399 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:21:04.365371 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ed378a_6a38_423b_a3d2_9c2622a65d42.slice/crio-98cee762af45dfe398d0da08b61f99a628e3ad4e871b53b81b1a99cb4f5e6679 WatchSource:0}: Error finding container 98cee762af45dfe398d0da08b61f99a628e3ad4e871b53b81b1a99cb4f5e6679: Status 404 returned error can't find the container with id 98cee762af45dfe398d0da08b61f99a628e3ad4e871b53b81b1a99cb4f5e6679 Nov 02 01:21:05.366978 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:05.366946 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" event={"ID":"d8ed378a-6a38-423b-a3d2-9c2622a65d42","Type":"ContainerStarted","Data":"98cee762af45dfe398d0da08b61f99a628e3ad4e871b53b81b1a99cb4f5e6679"} Nov 02 01:21:14.091321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.091287 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5ldp5"] Nov 02 01:21:14.091818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.091550 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" podUID="a72b5bff-447e-4df5-8519-9e91fda5346f" containerName="manager" containerID="cri-o://db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac" gracePeriod=30 Nov 02 01:21:14.093800 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.093774 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:21:14.360069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.360041 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:21:14.440971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.440939 4552 generic.go:356] "Generic (PLEG): container finished" podID="a72b5bff-447e-4df5-8519-9e91fda5346f" containerID="db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac" exitCode=0 Nov 02 01:21:14.441177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.440999 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" Nov 02 01:21:14.441177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.441031 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" event={"ID":"a72b5bff-447e-4df5-8519-9e91fda5346f","Type":"ContainerDied","Data":"db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac"} Nov 02 01:21:14.441177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.441092 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5ldp5" event={"ID":"a72b5bff-447e-4df5-8519-9e91fda5346f","Type":"ContainerDied","Data":"eb076197386067e3830c8c4387932d1c4bb22abc8bf8a829714ac2ff26042bf4"} Nov 02 01:21:14.441177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.441134 4552 scope.go:122] "RemoveContainer" containerID="db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac" Nov 02 01:21:14.451026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.451008 4552 scope.go:122] "RemoveContainer" containerID="db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac" Nov 02 01:21:14.451383 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:21:14.451354 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac\": container with ID starting with db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac not found: ID does not exist" containerID="db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac" Nov 02 01:21:14.451480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.451388 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac"} err="failed to get container status \"db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac\": rpc error: code = NotFound desc = could not find container \"db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac\": container with ID starting with db1bfe07adfbfd9fdd4ce859579a2f369e563e4aba8edcfdb15b003721bfddac not found: ID does not exist" Nov 02 01:21:14.505787 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.505754 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd\" (UniqueName: \"kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd\") pod \"a72b5bff-447e-4df5-8519-9e91fda5346f\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " Nov 02 01:21:14.505981 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.505808 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs\") pod \"a72b5bff-447e-4df5-8519-9e91fda5346f\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " Nov 02 01:21:14.505981 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.505932 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert\") pod \"a72b5bff-447e-4df5-8519-9e91fda5346f\" (UID: \"a72b5bff-447e-4df5-8519-9e91fda5346f\") " Nov 02 01:21:14.509958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.509920 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert" pod "a72b5bff-447e-4df5-8519-9e91fda5346f" (UID: "a72b5bff-447e-4df5-8519-9e91fda5346f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:21:14.510090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.510037 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs" pod "a72b5bff-447e-4df5-8519-9e91fda5346f" (UID: "a72b5bff-447e-4df5-8519-9e91fda5346f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:21:14.510826 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.510802 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd" pod "a72b5bff-447e-4df5-8519-9e91fda5346f" (UID: "a72b5bff-447e-4df5-8519-9e91fda5346f"). InnerVolumeSpecName "kube-api-access-x6lzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:21:14.608300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.608185 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6lzd\" (UniqueName: \"kubernetes.io/projected/a72b5bff-447e-4df5-8519-9e91fda5346f-kube-api-access-x6lzd\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:14.608300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.608228 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:14.608300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.608243 4552 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a72b5bff-447e-4df5-8519-9e91fda5346f-webhook-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:14.652877 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.652838 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kwxtx"] Nov 02 01:21:14.653687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.653668 4552 cpu_manager.go:414] "containerMap: removing container" podUID="a72b5bff-447e-4df5-8519-9e91fda5346f" containerName="manager" Nov 02 01:21:14.653834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.653794 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72b5bff-447e-4df5-8519-9e91fda5346f" containerName="manager" Nov 02 01:21:14.653960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.653948 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72b5bff-447e-4df5-8519-9e91fda5346f" containerName="manager" Nov 02 01:21:14.659015 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.658994 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.670027 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.669951 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kwxtx"] Nov 02 01:21:14.670762 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.670743 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 01:21:14.680656 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.670818 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-b6hsq\"" Nov 02 01:21:14.699278 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.699245 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s8xz5"] Nov 02 01:21:14.704843 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.704817 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.719817 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.719771 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-24xtf\"" Nov 02 01:21:14.719942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.719901 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 01:21:14.723806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.723781 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s8xz5"] Nov 02 01:21:14.768319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.768282 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5ldp5"] Nov 02 01:21:14.771621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.771586 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5ldp5"] Nov 02 01:21:14.810018 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.809986 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d4c10d-b4c8-4973-ba7f-56b3e857d4b5-cert\") pod \"kserve-controller-manager-776cc66d47-kwxtx\" (UID: \"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.810202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.810048 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdm6\" (UniqueName: \"kubernetes.io/projected/14d4c10d-b4c8-4973-ba7f-56b3e857d4b5-kube-api-access-hrdm6\") pod \"kserve-controller-manager-776cc66d47-kwxtx\" (UID: \"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.810202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.810089 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwcm\" (UniqueName: \"kubernetes.io/projected/7f972685-30f2-4bcc-9770-0c3a344d0dc7-kube-api-access-vkwcm\") pod \"model-serving-api-75bd89fdfc-s8xz5\" (UID: \"7f972685-30f2-4bcc-9770-0c3a344d0dc7\") " pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.810202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.810130 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f972685-30f2-4bcc-9770-0c3a344d0dc7-tls-certs\") pod \"model-serving-api-75bd89fdfc-s8xz5\" (UID: \"7f972685-30f2-4bcc-9770-0c3a344d0dc7\") " pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.911379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.911303 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d4c10d-b4c8-4973-ba7f-56b3e857d4b5-cert\") pod \"kserve-controller-manager-776cc66d47-kwxtx\" (UID: \"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.911379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.911356 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdm6\" (UniqueName: \"kubernetes.io/projected/14d4c10d-b4c8-4973-ba7f-56b3e857d4b5-kube-api-access-hrdm6\") pod \"kserve-controller-manager-776cc66d47-kwxtx\" (UID: \"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.911597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.911397 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwcm\" (UniqueName: \"kubernetes.io/projected/7f972685-30f2-4bcc-9770-0c3a344d0dc7-kube-api-access-vkwcm\") pod \"model-serving-api-75bd89fdfc-s8xz5\" (UID: \"7f972685-30f2-4bcc-9770-0c3a344d0dc7\") " pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.911597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.911429 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f972685-30f2-4bcc-9770-0c3a344d0dc7-tls-certs\") pod \"model-serving-api-75bd89fdfc-s8xz5\" (UID: \"7f972685-30f2-4bcc-9770-0c3a344d0dc7\") " pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.914317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.914292 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f972685-30f2-4bcc-9770-0c3a344d0dc7-tls-certs\") pod \"model-serving-api-75bd89fdfc-s8xz5\" (UID: \"7f972685-30f2-4bcc-9770-0c3a344d0dc7\") " pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.914467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.914450 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d4c10d-b4c8-4973-ba7f-56b3e857d4b5-cert\") pod \"kserve-controller-manager-776cc66d47-kwxtx\" (UID: \"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.929042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.929018 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwcm\" (UniqueName: \"kubernetes.io/projected/7f972685-30f2-4bcc-9770-0c3a344d0dc7-kube-api-access-vkwcm\") pod \"model-serving-api-75bd89fdfc-s8xz5\" (UID: \"7f972685-30f2-4bcc-9770-0c3a344d0dc7\") " pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:14.929193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.929069 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdm6\" (UniqueName: \"kubernetes.io/projected/14d4c10d-b4c8-4973-ba7f-56b3e857d4b5-kube-api-access-hrdm6\") pod \"kserve-controller-manager-776cc66d47-kwxtx\" (UID: \"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:14.979635 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:14.979595 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:15.030571 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:15.030535 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:15.311954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:15.311921 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kwxtx"] Nov 02 01:21:15.313531 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:21:15.313482 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d4c10d_b4c8_4973_ba7f_56b3e857d4b5.slice/crio-7a903861d10af9c0fd4e2f818b7167a8fc2d6fba7a9fedce5779e2e58dcae66e WatchSource:0}: Error finding container 7a903861d10af9c0fd4e2f818b7167a8fc2d6fba7a9fedce5779e2e58dcae66e: Status 404 returned error can't find the container with id 7a903861d10af9c0fd4e2f818b7167a8fc2d6fba7a9fedce5779e2e58dcae66e Nov 02 01:21:15.352532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:15.352499 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s8xz5"] Nov 02 01:21:15.448390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:15.448352 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" event={"ID":"7f972685-30f2-4bcc-9770-0c3a344d0dc7","Type":"ContainerStarted","Data":"63806d14dc3add908f92cd6c445c4edf00df457e936c0c7807d2f44d01451d5c"} Nov 02 01:21:15.450861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:15.450829 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" event={"ID":"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5","Type":"ContainerStarted","Data":"7a903861d10af9c0fd4e2f818b7167a8fc2d6fba7a9fedce5779e2e58dcae66e"} Nov 02 01:21:16.052926 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:16.052375 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72b5bff-447e-4df5-8519-9e91fda5346f" path="/var/lib/kubelet/pods/a72b5bff-447e-4df5-8519-9e91fda5346f/volumes" Nov 02 01:21:18.474812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:18.474691 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" event={"ID":"7f972685-30f2-4bcc-9770-0c3a344d0dc7","Type":"ContainerStarted","Data":"8f4cebcd48e12750f0522c3f1d98bb58f025d490f6b0d4f9a272cb9c437ad7d2"} Nov 02 01:21:18.474812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:18.474776 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:18.476337 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:18.476313 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" event={"ID":"d8ed378a-6a38-423b-a3d2-9c2622a65d42","Type":"ContainerStarted","Data":"e0b77fd56151c8fce1798a1f0c287fe06c9f59edae9078eb47d5480cc563406c"} Nov 02 01:21:18.476613 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:18.476375 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:18.495675 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:18.495634 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" podStartSLOduration=1.995356321 podStartE2EDuration="4.495617637s" totalImagesPullingTime="2.500261316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:18.493790733 +0000 UTC m=+378.599579648" watchObservedRunningTime="2026-11-02 01:21:18.495617637 +0000 UTC m=+378.601406550" Nov 02 01:21:18.517588 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:18.517538 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" podStartSLOduration=2.173772155 podStartE2EDuration="15.517524564s" totalImagesPullingTime="13.343752409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:18.515005716 +0000 UTC m=+378.620794631" watchObservedRunningTime="2026-11-02 01:21:18.517524564 +0000 UTC m=+378.623313478" Nov 02 01:21:19.483851 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:19.483809 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" event={"ID":"14d4c10d-b4c8-4973-ba7f-56b3e857d4b5","Type":"ContainerStarted","Data":"23c6e71460c0190ec933181d2b3f6af224033e16c8da59708ad98f52081cd0f4"} Nov 02 01:21:19.484372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:19.484024 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:19.502995 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:19.502939 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" podStartSLOduration=2.337167011 podStartE2EDuration="5.502923355s" totalImagesPullingTime="3.165756344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:19.502532165 +0000 UTC m=+379.608321078" watchObservedRunningTime="2026-11-02 01:21:19.502923355 +0000 UTC m=+379.608712273" Nov 02 01:21:27.750039 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:21:27.750006 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:21:29.489206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:29.489172 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-s8xz5" Nov 02 01:21:29.490172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:29.490147 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rx7hr" Nov 02 01:21:32.306532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:32.306496 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:21:41.612789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:41.612759 4552 generic.go:356] "Generic (PLEG): container finished" podID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerID="f0efbb0afd5f91e982642bc8edf075c92c3626c3ad7edf9ef0eb1dee84961ad1" exitCode=0 Nov 02 01:21:41.613276 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:41.612834 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" event={"ID":"a9f2a834-c86c-4af6-bbde-17ba9e356e2d","Type":"ContainerDied","Data":"f0efbb0afd5f91e982642bc8edf075c92c3626c3ad7edf9ef0eb1dee84961ad1"} Nov 02 01:21:41.613276 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:41.613240 4552 scope.go:122] "RemoveContainer" containerID="f0efbb0afd5f91e982642bc8edf075c92c3626c3ad7edf9ef0eb1dee84961ad1" Nov 02 01:21:41.614475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:41.614443 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:21:41.614590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:41.614531 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:21:41.614590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:41.614580 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:21:42.168968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.168935 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-427qz"] Nov 02 01:21:42.177998 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.177974 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.181946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.181902 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-427qz"] Nov 02 01:21:42.182187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.182165 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 01:21:42.182401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.182373 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 01:21:42.182976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.182958 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\"" Nov 02 01:21:42.194958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.194940 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 01:21:42.195059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.195005 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 01:21:42.278172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.278099 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.278355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.278184 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.278355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.278249 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.278355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.278337 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkx7\" (UniqueName: \"kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.379803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.379773 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.379999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.379814 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.379999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.379838 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.379999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.379870 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkx7\" (UniqueName: \"kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.380494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.380470 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.382329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.382302 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.382433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.382411 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.395755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.395726 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkx7\" (UniqueName: \"kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7\") pod \"maas-api-6cff7899-427qz\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.494011 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.493979 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:42.620796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.620765 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" event={"ID":"a9f2a834-c86c-4af6-bbde-17ba9e356e2d","Type":"ContainerStarted","Data":"cf29c45c77e8aeeee6561b837c2d419dbac46ee2e3926b91e367358a9bbf47d6"} Nov 02 01:21:42.794558 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:42.794532 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-427qz"] Nov 02 01:21:42.796044 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:21:42.796018 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1d7096_00de_46aa_969a_72a24fb54adb.slice/crio-526b2bc0f23a1e4682483bc5f2cf921e0846d195a37f30587d87f5d35b566335 WatchSource:0}: Error finding container 526b2bc0f23a1e4682483bc5f2cf921e0846d195a37f30587d87f5d35b566335: Status 404 returned error can't find the container with id 526b2bc0f23a1e4682483bc5f2cf921e0846d195a37f30587d87f5d35b566335 Nov 02 01:21:43.169067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:43.168983 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:21:43.169244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:43.169181 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 01:21:43.169293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:43.169231 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 01:21:43.629557 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:43.629520 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" event={"ID":"fb1d7096-00de-46aa-969a-72a24fb54adb","Type":"ContainerStarted","Data":"526b2bc0f23a1e4682483bc5f2cf921e0846d195a37f30587d87f5d35b566335"} Nov 02 01:21:44.169081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:44.169047 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 01:21:44.169286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:44.169109 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 01:21:45.168948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:45.168903 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 01:21:45.169431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:45.168969 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 01:21:45.670878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:45.670829 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" event={"ID":"fb1d7096-00de-46aa-969a-72a24fb54adb","Type":"ContainerStarted","Data":"5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932"} Nov 02 01:21:45.671068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:45.670899 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:45.691488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:45.691437 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" podStartSLOduration=1.818683058 podStartE2EDuration="3.69142118s" totalImagesPullingTime="1.872738122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:45.689822698 +0000 UTC m=+405.795611616" watchObservedRunningTime="2026-11-02 01:21:45.69142118 +0000 UTC m=+405.797210093" Nov 02 01:21:46.169520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:46.169477 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 01:21:46.169941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:46.169536 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 01:21:47.169566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:47.169517 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 01:21:47.170016 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:47.169589 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 01:21:48.173522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:48.173492 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:21:48.173904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:48.173874 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:21:48.174604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:48.174587 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" Nov 02 01:21:50.495320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:50.495291 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kwxtx" Nov 02 01:21:51.681568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:21:51.681539 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 01:21:57.750110 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:21:57.750072 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:22:27.749796 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:22:27.749756 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:22:57.748590 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:22:57.748545 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:23:04.444476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.444444 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7"] Nov 02 01:23:04.448435 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.448411 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.457299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.457279 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 01:23:04.458216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.457457 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 01:23:04.458350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.458326 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 01:23:04.458425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.458330 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5855b\"" Nov 02 01:23:04.461505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.461485 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7"] Nov 02 01:23:04.634407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.634371 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.634585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.634427 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.634630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.634598 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4k5\" (UniqueName: \"kubernetes.io/projected/5fe9f839-921c-433a-a857-9532cae45e2b-kube-api-access-2z4k5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.634671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.634639 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.634704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.634668 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe9f839-921c-433a-a857-9532cae45e2b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.634792 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.634780 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.735521 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.735480 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4k5\" (UniqueName: \"kubernetes.io/projected/5fe9f839-921c-433a-a857-9532cae45e2b-kube-api-access-2z4k5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.735707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.735530 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.735707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.735550 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe9f839-921c-433a-a857-9532cae45e2b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.735707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.735580 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.735707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.735600 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.735707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.735624 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.736080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.736050 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.736080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.736057 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.736080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.736072 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.738070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.738043 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe9f839-921c-433a-a857-9532cae45e2b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.738297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.738276 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe9f839-921c-433a-a857-9532cae45e2b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.747754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.747729 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4k5\" (UniqueName: \"kubernetes.io/projected/5fe9f839-921c-433a-a857-9532cae45e2b-kube-api-access-2z4k5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7qsc7\" (UID: \"5fe9f839-921c-433a-a857-9532cae45e2b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.765327 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.765306 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:04.872625 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:04.872595 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:23:05.052877 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:05.052845 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7"] Nov 02 01:23:05.055864 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:05.055836 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe9f839_921c_433a_a857_9532cae45e2b.slice/crio-8fc6e9e4c54d5422ff80d6944c9afb814dba9c55211afadaee0fa4059d13e335 WatchSource:0}: Error finding container 8fc6e9e4c54d5422ff80d6944c9afb814dba9c55211afadaee0fa4059d13e335: Status 404 returned error can't find the container with id 8fc6e9e4c54d5422ff80d6944c9afb814dba9c55211afadaee0fa4059d13e335 Nov 02 01:23:05.111965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:05.111925 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" event={"ID":"5fe9f839-921c-433a-a857-9532cae45e2b","Type":"ContainerStarted","Data":"8fc6e9e4c54d5422ff80d6944c9afb814dba9c55211afadaee0fa4059d13e335"} Nov 02 01:23:06.824679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:06.824640 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:23:06.982891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:06.982845 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x"] Nov 02 01:23:06.986764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:06.986740 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:06.992680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:06.992656 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 01:23:07.002423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.002390 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x"] Nov 02 01:23:07.127350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.127267 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" event={"ID":"5fe9f839-921c-433a-a857-9532cae45e2b","Type":"ContainerStarted","Data":"055783a68463c31968a4a93ae4fc6ad43ecb7d018da35945daff5e6377f83d3e"} Nov 02 01:23:07.159260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.153111 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" podStartSLOduration=1.835856141 podStartE2EDuration="3.153090049s" totalImagesPullingTime="1.317233908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:07.150179472 +0000 UTC m=+487.255968388" watchObservedRunningTime="2026-11-02 01:23:07.153090049 +0000 UTC m=+487.258878963" Nov 02 01:23:07.161350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.160224 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31aa41f-6bdd-4401-b45b-8a7e863305ae-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.161350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.160359 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.161350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.160427 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.161350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.160614 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.161350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.160653 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989wt\" (UniqueName: \"kubernetes.io/projected/a31aa41f-6bdd-4401-b45b-8a7e863305ae-kube-api-access-989wt\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.161350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.160790 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.261428 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989wt\" (UniqueName: \"kubernetes.io/projected/a31aa41f-6bdd-4401-b45b-8a7e863305ae-kube-api-access-989wt\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.261506 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.261540 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31aa41f-6bdd-4401-b45b-8a7e863305ae-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.261577 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.261620 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.261692 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.262047 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.262433 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.263692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.263427 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.265498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.265472 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31aa41f-6bdd-4401-b45b-8a7e863305ae-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.266423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.266401 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31aa41f-6bdd-4401-b45b-8a7e863305ae-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.275279 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.275252 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-989wt\" (UniqueName: \"kubernetes.io/projected/a31aa41f-6bdd-4401-b45b-8a7e863305ae-kube-api-access-989wt\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kb25x\" (UID: \"a31aa41f-6bdd-4401-b45b-8a7e863305ae\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.301996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.301971 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:07.593730 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.593678 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x"] Nov 02 01:23:07.595105 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:07.595071 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31aa41f_6bdd_4401_b45b_8a7e863305ae.slice/crio-14681aa8c83c5dd51ad85e2e86b2fd4105b0cef32bcfeb2c81a3bb8c8c6b93b5 WatchSource:0}: Error finding container 14681aa8c83c5dd51ad85e2e86b2fd4105b0cef32bcfeb2c81a3bb8c8c6b93b5: Status 404 returned error can't find the container with id 14681aa8c83c5dd51ad85e2e86b2fd4105b0cef32bcfeb2c81a3bb8c8c6b93b5 Nov 02 01:23:07.765952 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:07.765909 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:08.135828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:08.135729 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" event={"ID":"a31aa41f-6bdd-4401-b45b-8a7e863305ae","Type":"ContainerStarted","Data":"0e7b440ec2ac2a756e4a340a8df4d0c4a6e6eda737c6a7f5245ac433845b3218"} Nov 02 01:23:08.135828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:08.135797 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" event={"ID":"a31aa41f-6bdd-4401-b45b-8a7e863305ae","Type":"ContainerStarted","Data":"14681aa8c83c5dd51ad85e2e86b2fd4105b0cef32bcfeb2c81a3bb8c8c6b93b5"} Nov 02 01:23:08.156975 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:08.156922 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" podStartSLOduration=1.873571892 podStartE2EDuration="2.156908531s" totalImagesPullingTime="283.336639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:08.154769564 +0000 UTC m=+488.260558495" watchObservedRunningTime="2026-11-02 01:23:08.156908531 +0000 UTC m=+488.262697441" Nov 02 01:23:08.302643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:08.302607 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:09.586934 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:09.586855 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:23:10.610184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.610150 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th"] Nov 02 01:23:10.614422 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.614397 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.624019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.623999 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 01:23:10.626272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.626245 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th"] Nov 02 01:23:10.695028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.694986 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.695028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.695024 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.695314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.695057 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82568ce-5132-457b-8166-ec3b93c94297-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.695314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.695200 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.695314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.695262 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.695314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.695279 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfn4\" (UniqueName: \"kubernetes.io/projected/d82568ce-5132-457b-8166-ec3b93c94297-kube-api-access-wjfn4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.795989 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796086 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfn4\" (UniqueName: \"kubernetes.io/projected/d82568ce-5132-457b-8166-ec3b93c94297-kube-api-access-wjfn4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796231 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796550 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796440 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796549 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796513 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796582 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82568ce-5132-457b-8166-ec3b93c94297-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796632 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.796861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.796800 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.799145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.799085 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82568ce-5132-457b-8166-ec3b93c94297-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.799855 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.799826 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82568ce-5132-457b-8166-ec3b93c94297-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.810333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.810312 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfn4\" (UniqueName: \"kubernetes.io/projected/d82568ce-5132-457b-8166-ec3b93c94297-kube-api-access-wjfn4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th\" (UID: \"d82568ce-5132-457b-8166-ec3b93c94297\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:10.930839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:10.930758 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:11.280821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:11.280793 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th"] Nov 02 01:23:11.282708 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:11.282671 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82568ce_5132_457b_8166_ec3b93c94297.slice/crio-2fd186727007599c28b4fcbb3d34d579506cb5145d38b1536ff80103a544b223 WatchSource:0}: Error finding container 2fd186727007599c28b4fcbb3d34d579506cb5145d38b1536ff80103a544b223: Status 404 returned error can't find the container with id 2fd186727007599c28b4fcbb3d34d579506cb5145d38b1536ff80103a544b223 Nov 02 01:23:12.167909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.167835 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" event={"ID":"d82568ce-5132-457b-8166-ec3b93c94297","Type":"ContainerStarted","Data":"6c406c9b7332fc29b64912fd584cfa57c50423cecb3aa15bfc7424130868f190"} Nov 02 01:23:12.170159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.170128 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" event={"ID":"d82568ce-5132-457b-8166-ec3b93c94297","Type":"ContainerStarted","Data":"2fd186727007599c28b4fcbb3d34d579506cb5145d38b1536ff80103a544b223"} Nov 02 01:23:12.191886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.191818 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" podStartSLOduration=1.9236318319999999 podStartE2EDuration="2.191798785s" totalImagesPullingTime="268.166953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:12.18881865 +0000 UTC m=+492.294607574" watchObservedRunningTime="2026-11-02 01:23:12.191798785 +0000 UTC m=+492.297587703" Nov 02 01:23:12.403091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.403002 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm"] Nov 02 01:23:12.408409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.408385 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.416918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.416883 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm"] Nov 02 01:23:12.420958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.420620 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 01:23:12.453080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.453040 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:23:12.514779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.514742 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.514954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.514797 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.514954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.514883 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjxc\" (UniqueName: \"kubernetes.io/projected/be4f2277-8eb3-42bb-8384-64bf743cb7aa-kube-api-access-ptjxc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.514954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.514937 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4f2277-8eb3-42bb-8384-64bf743cb7aa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.515074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.515051 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.515174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.515159 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.615976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.615932 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616004 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616031 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616062 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjxc\" (UniqueName: \"kubernetes.io/projected/be4f2277-8eb3-42bb-8384-64bf743cb7aa-kube-api-access-ptjxc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616156 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4f2277-8eb3-42bb-8384-64bf743cb7aa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616213 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616454 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616493 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.616586 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.616542 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.618601 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.618575 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4f2277-8eb3-42bb-8384-64bf743cb7aa-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.618894 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.618879 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4f2277-8eb3-42bb-8384-64bf743cb7aa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.628351 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.628331 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjxc\" (UniqueName: \"kubernetes.io/projected/be4f2277-8eb3-42bb-8384-64bf743cb7aa-kube-api-access-ptjxc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm\" (UID: \"be4f2277-8eb3-42bb-8384-64bf743cb7aa\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.728104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.728072 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:12.931868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:12.931838 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:13.015910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:13.015870 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm"] Nov 02 01:23:13.017084 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:13.017059 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4f2277_8eb3_42bb_8384_64bf743cb7aa.slice/crio-115746d99e4aea21c88c2e74717e33e137bc834f3200ba91baa7d5c64c889b77 WatchSource:0}: Error finding container 115746d99e4aea21c88c2e74717e33e137bc834f3200ba91baa7d5c64c889b77: Status 404 returned error can't find the container with id 115746d99e4aea21c88c2e74717e33e137bc834f3200ba91baa7d5c64c889b77 Nov 02 01:23:13.175858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:13.175816 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" event={"ID":"be4f2277-8eb3-42bb-8384-64bf743cb7aa","Type":"ContainerStarted","Data":"115746d99e4aea21c88c2e74717e33e137bc834f3200ba91baa7d5c64c889b77"} Nov 02 01:23:14.187395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:14.187357 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" event={"ID":"be4f2277-8eb3-42bb-8384-64bf743cb7aa","Type":"ContainerStarted","Data":"a75699414d3006187bac445537508dae1c3c25089b2bfb22c2d18ae881230468"} Nov 02 01:23:14.209600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:14.209543 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" podStartSLOduration=1.930701668 podStartE2EDuration="2.209528471s" totalImagesPullingTime="278.826803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:14.207097793 +0000 UTC m=+494.312886709" watchObservedRunningTime="2026-11-02 01:23:14.209528471 +0000 UTC m=+494.315317415" Nov 02 01:23:14.729195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:14.729161 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:14.765846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:14.765801 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:14.782651 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:14.782617 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:15.205129 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:15.205026 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7qsc7" Nov 02 01:23:17.302580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:17.302543 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:17.315861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:17.315837 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:18.153482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:18.153449 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:23:18.223739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:18.223710 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" Nov 02 01:23:20.931749 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:20.931707 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:20.944743 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:20.944715 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:21.242190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:21.242161 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th" Nov 02 01:23:22.728556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:22.728525 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:22.740828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:22.740808 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:23.186509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.186428 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q"] Nov 02 01:23:23.190814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.190790 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.207829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.207802 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q"] Nov 02 01:23:23.255987 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.255963 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm" Nov 02 01:23:23.329613 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329575 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.329813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329629 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.329813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329745 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dp6s\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.329940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329823 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.329940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329871 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.329940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329895 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.330076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.329977 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.330076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.330028 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.330225 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.330210 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.431929 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432027 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432068 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dp6s\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432098 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432184 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432220 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432265 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432302 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.432858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432405 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.433453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.432940 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.433453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.433170 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.436877 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.436805 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.439375 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.439348 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.440014 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.439985 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.440315 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.440086 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.440440 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.440333 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.445780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.445756 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.445922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.445905 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dp6s\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s\") pod \"maas-default-gateway-openshift-default-76597d6855-l6m5q\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.510269 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.510221 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:23.792688 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.792648 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q"] Nov 02 01:23:23.793552 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:23.793526 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53142e56_8cd2_46b3_a11d_18245c3f849d.slice/crio-686b24ad51cb1d4d10bff7af6ac07154e39540c5f783a2e902826a35795834b3 WatchSource:0}: Error finding container 686b24ad51cb1d4d10bff7af6ac07154e39540c5f783a2e902826a35795834b3: Status 404 returned error can't find the container with id 686b24ad51cb1d4d10bff7af6ac07154e39540c5f783a2e902826a35795834b3 Nov 02 01:23:23.796027 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.795990 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:23.796104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.796065 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:23.796104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:23.796097 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:24.270747 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:24.270706 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" event={"ID":"53142e56-8cd2-46b3-a11d-18245c3f849d","Type":"ContainerStarted","Data":"213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb"} Nov 02 01:23:24.270939 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:24.270767 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" event={"ID":"53142e56-8cd2-46b3-a11d-18245c3f849d","Type":"ContainerStarted","Data":"686b24ad51cb1d4d10bff7af6ac07154e39540c5f783a2e902826a35795834b3"} Nov 02 01:23:24.298699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:24.298646 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podStartSLOduration=1.298632642 podStartE2EDuration="1.298632642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:24.295018852 +0000 UTC m=+504.400807768" watchObservedRunningTime="2026-11-02 01:23:24.298632642 +0000 UTC m=+504.404421555" Nov 02 01:23:24.510733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:24.510694 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:24.512295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:24.512269 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:24.512397 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:24.512334 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:25.511617 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:25.511576 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:25.512046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:25.511638 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:26.510974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:26.510935 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:26.511193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:26.510999 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:27.511476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:27.511433 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:27.511944 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:27.511500 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:27.773503 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:23:27.773414 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:23:28.510796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:28.510757 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:28.510961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:28.510812 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:29.511593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:29.511555 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:29.511988 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:29.511613 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:30.511162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:30.511100 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-l6m5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 01:23:30.511385 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:30.511188 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 01:23:31.992742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:31.992713 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:31.993243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:31.993134 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:31.993858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:31.993840 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:23:38.209257 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.209221 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm"] Nov 02 01:23:38.214333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.214308 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.231935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.231894 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm"] Nov 02 01:23:38.313875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.313836 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.313875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.313890 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36874ab-fccd-45b7-9497-668a0fd1c983-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314002 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314075 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314179 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314218 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314388 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314320 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314388 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314351 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.314388 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.314382 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phhd\" (UniqueName: \"kubernetes.io/projected/a36874ab-fccd-45b7-9497-668a0fd1c983-kube-api-access-7phhd\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.414914 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.414871 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.414914 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.414914 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.414934 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.414978 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.414995 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415012 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phhd\" (UniqueName: \"kubernetes.io/projected/a36874ab-fccd-45b7-9497-668a0fd1c983-kube-api-access-7phhd\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415035 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415057 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36874ab-fccd-45b7-9497-668a0fd1c983-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415109 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415570 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415398 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415570 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415488 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415570 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415536 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.415726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.415627 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.416281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.416259 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36874ab-fccd-45b7-9497-668a0fd1c983-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.417488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.417468 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.417709 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.417675 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.426925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.426900 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36874ab-fccd-45b7-9497-668a0fd1c983-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.427031 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.427016 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phhd\" (UniqueName: \"kubernetes.io/projected/a36874ab-fccd-45b7-9497-668a0fd1c983-kube-api-access-7phhd\") pod \"maas-default-gateway-openshift-default-76597d6855-p24rm\" (UID: \"a36874ab-fccd-45b7-9497-668a0fd1c983\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.546491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.546455 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:38.827361 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.827334 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm"] Nov 02 01:23:38.829962 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:38.829932 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36874ab_fccd_45b7_9497_668a0fd1c983.slice/crio-cef26ed5cc730ed2ad55972efa2cd6bff1f8b7720fb21896274fe50a4737a8ba WatchSource:0}: Error finding container cef26ed5cc730ed2ad55972efa2cd6bff1f8b7720fb21896274fe50a4737a8ba: Status 404 returned error can't find the container with id cef26ed5cc730ed2ad55972efa2cd6bff1f8b7720fb21896274fe50a4737a8ba Nov 02 01:23:38.832470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.832433 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:38.832576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.832515 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:38.832576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:38.832558 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:39.369971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:39.369931 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" event={"ID":"a36874ab-fccd-45b7-9497-668a0fd1c983","Type":"ContainerStarted","Data":"48e4c9fbdddccd300ef0c99b231387c10759de3291648bd647b6bc01fa6955fa"} Nov 02 01:23:39.370422 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:39.369992 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" event={"ID":"a36874ab-fccd-45b7-9497-668a0fd1c983","Type":"ContainerStarted","Data":"cef26ed5cc730ed2ad55972efa2cd6bff1f8b7720fb21896274fe50a4737a8ba"} Nov 02 01:23:39.395490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:39.395447 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podStartSLOduration=1.395433694 podStartE2EDuration="1.395433694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:39.393014965 +0000 UTC m=+519.498803879" watchObservedRunningTime="2026-11-02 01:23:39.395433694 +0000 UTC m=+519.501222677" Nov 02 01:23:39.562144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:39.561315 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:39.562978 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:39.562948 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:39.563078 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:39.563011 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:40.561711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:40.561232 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:40.561711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:40.561298 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:41.557143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:41.556538 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:41.557143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:41.556617 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:42.558144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:42.557526 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:42.558144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:42.557592 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:43.558145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:43.555546 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:43.558145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:43.555615 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:44.553581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:44.553538 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:44.555507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:44.555470 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:45.547271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:45.547235 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:23:45.547827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:45.547288 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 01:23:47.109043 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:47.108993 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:47.109890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:47.109853 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:47.110446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:47.110415 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" Nov 02 01:23:50.635642 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:50.635597 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:23:52.932181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:52.932137 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-wvrnr"] Nov 02 01:23:52.940764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:52.940731 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-wvrnr"] Nov 02 01:23:52.952141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:52.941053 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:52.969645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:52.968991 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 01:23:53.004567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.004527 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 01:23:53.004932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.004912 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-cb6fd\"" Nov 02 01:23:53.042431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.041658 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.042431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.041778 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtgp\" (UniqueName: \"kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.143153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.142700 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtgp\" (UniqueName: \"kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.143153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.142817 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.143153 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:23:53.142996 4552 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Nov 02 01:23:53.143153 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:23:53.143068 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs podName:4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b nodeName:}" failed. No retries permitted until 2026-11-02 01:23:53.643044182 +0000 UTC m=+533.748833099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs") pod "ai-gateway-controller-c7987c6b6-wvrnr" (UID: "4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b") : secret "ai-gateway-controller-metrics-tls" not found Nov 02 01:23:53.244148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.233295 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtgp\" (UniqueName: \"kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.244148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.238773 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k"] Nov 02 01:23:53.268641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.268610 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.341797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.341763 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k"] Nov 02 01:23:53.371150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.366866 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.371611 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.371579 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5f593a-2283-4295-acd1-909ae2384f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.377154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.371880 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng7j\" (UniqueName: \"kubernetes.io/projected/7f5f593a-2283-4295-acd1-909ae2384f98-kube-api-access-xng7j\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.377444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.377418 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5f593a-2283-4295-acd1-909ae2384f98-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.377590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.377572 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.377753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.377735 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5f593a-2283-4295-acd1-909ae2384f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.377885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.377868 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.378015 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.377998 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.378203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.378185 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.491885 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5f593a-2283-4295-acd1-909ae2384f98-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.491948 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492003 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5f593a-2283-4295-acd1-909ae2384f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492032 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492070 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492141 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492193 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492230 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5f593a-2283-4295-acd1-909ae2384f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.492641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.492364 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng7j\" (UniqueName: \"kubernetes.io/projected/7f5f593a-2283-4295-acd1-909ae2384f98-kube-api-access-xng7j\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.493307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.493282 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.495399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.494158 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5f593a-2283-4295-acd1-909ae2384f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.495534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.494731 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.495645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.494962 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.512141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.497236 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.517795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.517734 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5f593a-2283-4295-acd1-909ae2384f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.533547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.532149 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5f593a-2283-4295-acd1-909ae2384f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.534453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.534400 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng7j\" (UniqueName: \"kubernetes.io/projected/7f5f593a-2283-4295-acd1-909ae2384f98-kube-api-access-xng7j\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.535068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.535021 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5f593a-2283-4295-acd1-909ae2384f98-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-q856k\" (UID: \"7f5f593a-2283-4295-acd1-909ae2384f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.695145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.694809 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.707145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.703870 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-wvrnr\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:53.749382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.747987 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:53.955088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:53.955053 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:54.055505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.055475 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k"] Nov 02 01:23:54.056259 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:54.056232 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5f593a_2283_4295_acd1_909ae2384f98.slice/crio-3ea7782a7032c5d5a6600ff7441913562631a8bbcbbe95a4ea46e18c3154641e WatchSource:0}: Error finding container 3ea7782a7032c5d5a6600ff7441913562631a8bbcbbe95a4ea46e18c3154641e: Status 404 returned error can't find the container with id 3ea7782a7032c5d5a6600ff7441913562631a8bbcbbe95a4ea46e18c3154641e Nov 02 01:23:54.058552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.058519 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:54.058657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.058584 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:54.058657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.058614 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:23:54.245449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.244331 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-wvrnr"] Nov 02 01:23:54.247254 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:54.247221 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd8fdcf_d9e0_435c_baca_6e9a1d7e043b.slice/crio-50b474d43b364f7e97b2fa4ceac863cc88043586dfe2dce3265ed45317a0c5f7 WatchSource:0}: Error finding container 50b474d43b364f7e97b2fa4ceac863cc88043586dfe2dce3265ed45317a0c5f7: Status 404 returned error can't find the container with id 50b474d43b364f7e97b2fa4ceac863cc88043586dfe2dce3265ed45317a0c5f7 Nov 02 01:23:54.544262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.544226 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" event={"ID":"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b","Type":"ContainerStarted","Data":"50b474d43b364f7e97b2fa4ceac863cc88043586dfe2dce3265ed45317a0c5f7"} Nov 02 01:23:54.545897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.545868 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" event={"ID":"7f5f593a-2283-4295-acd1-909ae2384f98","Type":"ContainerStarted","Data":"e94aa1c2a74c39b1b2148e2d7f90383edcc3c95f3ae41e118ced4597f88bf48a"} Nov 02 01:23:54.546019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.545919 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" event={"ID":"7f5f593a-2283-4295-acd1-909ae2384f98","Type":"ContainerStarted","Data":"3ea7782a7032c5d5a6600ff7441913562631a8bbcbbe95a4ea46e18c3154641e"} Nov 02 01:23:54.609089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.609022 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podStartSLOduration=1.6089940459999998 podStartE2EDuration="1.608994046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:54.606029539 +0000 UTC m=+534.711818453" watchObservedRunningTime="2026-11-02 01:23:54.608994046 +0000 UTC m=+534.714782959" Nov 02 01:23:54.749252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:54.749211 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:23:55.765281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:55.765240 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:23:55.765695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:55.765309 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:23:56.760975 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:56.760936 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:23:56.761293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:56.761260 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:23:57.589936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:57.589902 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" event={"ID":"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b","Type":"ContainerStarted","Data":"37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207"} Nov 02 01:23:57.590369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:57.590045 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:23:57.611147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:57.611062 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" podStartSLOduration=2.951439365 podStartE2EDuration="5.611046946s" totalImagesPullingTime="2.659607581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:57.608187094 +0000 UTC m=+537.713976000" watchObservedRunningTime="2026-11-02 01:23:57.611046946 +0000 UTC m=+537.716835859" Nov 02 01:23:57.751604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:57.751562 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:23:57.751915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:57.751880 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:23:57.774761 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:23:57.774719 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:23:58.751554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:58.751514 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:23:58.752192 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:58.752158 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:23:58.971208 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:58.971173 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-s25kq"] Nov 02 01:23:58.976867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:58.976839 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-s25kq"] Nov 02 01:23:58.976992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:58.976964 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.057982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.057849 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bbqgh\"" Nov 02 01:23:59.058222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.057892 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 01:23:59.076904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.076858 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d8235e-3957-42a3-ae40-7bb53eea23f8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-s25kq\" (UID: \"81d8235e-3957-42a3-ae40-7bb53eea23f8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.077232 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.077216 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4k4q\" (UniqueName: \"kubernetes.io/projected/81d8235e-3957-42a3-ae40-7bb53eea23f8-kube-api-access-d4k4q\") pod \"payload-processing-55fbbbf9dc-s25kq\" (UID: \"81d8235e-3957-42a3-ae40-7bb53eea23f8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.178823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.178788 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d8235e-3957-42a3-ae40-7bb53eea23f8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-s25kq\" (UID: \"81d8235e-3957-42a3-ae40-7bb53eea23f8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.179043 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.178865 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4k4q\" (UniqueName: \"kubernetes.io/projected/81d8235e-3957-42a3-ae40-7bb53eea23f8-kube-api-access-d4k4q\") pod \"payload-processing-55fbbbf9dc-s25kq\" (UID: \"81d8235e-3957-42a3-ae40-7bb53eea23f8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.179789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.179768 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d8235e-3957-42a3-ae40-7bb53eea23f8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-s25kq\" (UID: \"81d8235e-3957-42a3-ae40-7bb53eea23f8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.191255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.191234 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4k4q\" (UniqueName: \"kubernetes.io/projected/81d8235e-3957-42a3-ae40-7bb53eea23f8-kube-api-access-d4k4q\") pod \"payload-processing-55fbbbf9dc-s25kq\" (UID: \"81d8235e-3957-42a3-ae40-7bb53eea23f8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.292580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.292536 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:23:59.608452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.606433 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-s25kq"] Nov 02 01:23:59.615135 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:23:59.609384 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d8235e_3957_42a3_ae40_7bb53eea23f8.slice/crio-4256b122235a16e2b1bb606bb8e4c0ed9f516af0446489cbeb63f93d589e31db WatchSource:0}: Error finding container 4256b122235a16e2b1bb606bb8e4c0ed9f516af0446489cbeb63f93d589e31db: Status 404 returned error can't find the container with id 4256b122235a16e2b1bb606bb8e4c0ed9f516af0446489cbeb63f93d589e31db Nov 02 01:23:59.755143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.751572 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:23:59.755143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:23:59.751642 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:24:00.620140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:00.619993 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" event={"ID":"81d8235e-3957-42a3-ae40-7bb53eea23f8","Type":"ContainerStarted","Data":"4256b122235a16e2b1bb606bb8e4c0ed9f516af0446489cbeb63f93d589e31db"} Nov 02 01:24:00.751160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:00.750379 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:24:00.751160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:00.750448 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:24:01.748953 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:01.748912 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 01:24:01.749427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:01.748985 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 01:24:02.637920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:02.637872 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" event={"ID":"81d8235e-3957-42a3-ae40-7bb53eea23f8","Type":"ContainerStarted","Data":"ba7ca6f15c04c68226123559025e84ace9008ead1e9b19669a7d2d337243a2a9"} Nov 02 01:24:02.638161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:02.638013 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:24:02.657890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:02.657830 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" podStartSLOduration=2.083666628 podStartE2EDuration="4.657809851s" totalImagesPullingTime="2.574143223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:02.65518588 +0000 UTC m=+542.760974795" watchObservedRunningTime="2026-11-02 01:24:02.657809851 +0000 UTC m=+542.763598764" Nov 02 01:24:03.791326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:03.791271 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:24:03.791849 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:03.791356 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:24:04.833335 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:04.833287 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-q856k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:24:04.833853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:04.833364 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" podUID="7f5f593a-2283-4295-acd1-909ae2384f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:24:04.950099 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:04.950054 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:24:04.950537 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:04.950502 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:24:04.951340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:04.951318 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-q856k" Nov 02 01:24:08.598793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:08.598766 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:24:13.646916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:13.646888 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-s25kq" Nov 02 01:24:23.358841 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.358806 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-wvrnr"] Nov 02 01:24:23.359284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.358999 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" podUID="4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" containerName="manager" containerID="cri-o://37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207" gracePeriod=30 Nov 02 01:24:23.616317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.616249 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:24:23.736479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.736428 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp\" (UniqueName: \"kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp\") pod \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " Nov 02 01:24:23.736670 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.736509 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\") pod \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\" (UID: \"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b\") " Nov 02 01:24:23.738740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.738707 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs" pod "4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" (UID: "4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:24:23.738872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.738754 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp" pod "4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" (UID: "4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b"). InnerVolumeSpecName "kube-api-access-dqtgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:24:23.774098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.774060 4552 generic.go:356] "Generic (PLEG): container finished" podID="4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" containerID="37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207" exitCode=0 Nov 02 01:24:23.774274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.774111 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" Nov 02 01:24:23.774274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.774145 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" event={"ID":"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b","Type":"ContainerDied","Data":"37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207"} Nov 02 01:24:23.774274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.774201 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-wvrnr" event={"ID":"4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b","Type":"ContainerDied","Data":"50b474d43b364f7e97b2fa4ceac863cc88043586dfe2dce3265ed45317a0c5f7"} Nov 02 01:24:23.774274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.774214 4552 scope.go:122] "RemoveContainer" containerID="37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207" Nov 02 01:24:23.784439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.784420 4552 scope.go:122] "RemoveContainer" containerID="37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207" Nov 02 01:24:23.784723 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:24:23.784702 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207\": container with ID starting with 37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207 not found: ID does not exist" containerID="37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207" Nov 02 01:24:23.784791 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.784732 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207"} err="failed to get container status \"37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207\": rpc error: code = NotFound desc = could not find container \"37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207\": container with ID starting with 37d98bf8ab9d88cdb6e589e4f7db471a2c73f9df9fa84012bfdb2c4b05c7e207 not found: ID does not exist" Nov 02 01:24:23.797794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.797765 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-wvrnr"] Nov 02 01:24:23.803290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.803264 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-wvrnr"] Nov 02 01:24:23.838303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.838275 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqtgp\" (UniqueName: \"kubernetes.io/projected/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-kube-api-access-dqtgp\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:24:23.838303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:23.838301 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:24:24.043590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:24:24.043557 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" path="/var/lib/kubelet/pods/4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b/volumes" Nov 02 01:24:27.749805 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:24:27.749761 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:24:57.750442 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:24:57.750403 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:25:13.597354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.597317 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv"] Nov 02 01:25:13.597836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.597821 4552 cpu_manager.go:414] "containerMap: removing container" podUID="4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" containerName="manager" Nov 02 01:25:13.597880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.597837 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" containerName="manager" Nov 02 01:25:13.597918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.597912 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd8fdcf-d9e0-435c-baca-6e9a1d7e043b" containerName="manager" Nov 02 01:25:13.600672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.600657 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.607871 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.607844 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-b9d816\"" Nov 02 01:25:13.608064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.608034 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-b9d816-dockercfg-l7j4r\"" Nov 02 01:25:13.612779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.612738 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv"] Nov 02 01:25:13.635919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.635881 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc"] Nov 02 01:25:13.639945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.639921 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.652741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.652711 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc"] Nov 02 01:25:13.665706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.665649 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.665875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.665784 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltl9\" (UniqueName: \"kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9\") pod \"payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.665927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.665872 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49xf\" (UniqueName: \"kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf\") pod \"payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.668288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.668262 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.769360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.769320 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.769556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.769371 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.769556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.769421 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltl9\" (UniqueName: \"kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9\") pod \"payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.769556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.769465 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49xf\" (UniqueName: \"kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf\") pod \"payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.770171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.770149 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.770215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.770161 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.782714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.782683 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49xf\" (UniqueName: \"kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf\") pod \"payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.782852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.782734 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltl9\" (UniqueName: \"kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9\") pod \"payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:13.916599 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.916495 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:13.956062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:13.956015 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:14.201993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.201957 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv"] Nov 02 01:25:14.202965 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:14.202928 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b58629a_2db8_4e59_a467_f9b597164194.slice/crio-a1b2ae250ca1a26b860a4e2c044ef3aaa6776f5150681ed11c7ccf7a54aefa5c WatchSource:0}: Error finding container a1b2ae250ca1a26b860a4e2c044ef3aaa6776f5150681ed11c7ccf7a54aefa5c: Status 404 returned error can't find the container with id a1b2ae250ca1a26b860a4e2c044ef3aaa6776f5150681ed11c7ccf7a54aefa5c Nov 02 01:25:14.253021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.252988 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc"] Nov 02 01:25:14.253259 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:14.253239 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f8a4bf_fc12_45ed_bb85_245fbf05bde4.slice/crio-60ef273828f4cc84917a5b3c4484d332ecefaa9e562193a4587a0e198c79ddd1 WatchSource:0}: Error finding container 60ef273828f4cc84917a5b3c4484d332ecefaa9e562193a4587a0e198c79ddd1: Status 404 returned error can't find the container with id 60ef273828f4cc84917a5b3c4484d332ecefaa9e562193a4587a0e198c79ddd1 Nov 02 01:25:14.278355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.278329 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp"] Nov 02 01:25:14.282150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.282107 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.288305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.288283 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5687bc-dockercfg-fjfnn\"" Nov 02 01:25:14.288305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.288286 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5687bc\"" Nov 02 01:25:14.293732 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.293710 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp"] Nov 02 01:25:14.345228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.345191 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8"] Nov 02 01:25:14.349735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.349709 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.356085 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.356059 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8"] Nov 02 01:25:14.379230 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.379195 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57pn\" (UniqueName: \"kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn\") pod \"payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.379416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.379253 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26n86\" (UniqueName: \"kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86\") pod \"payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.379416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.379377 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.379504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.379434 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.480762 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.480726 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57pn\" (UniqueName: \"kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn\") pod \"payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.480947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.480774 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26n86\" (UniqueName: \"kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86\") pod \"payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.480947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.480893 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.481066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.480946 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.481672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.481653 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.481732 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.481675 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.492989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.492963 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26n86\" (UniqueName: \"kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86\") pod \"payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.493140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.493020 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57pn\" (UniqueName: \"kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn\") pod \"payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.618780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.618746 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:14.630524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.630495 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg"] Nov 02 01:25:14.635199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.634659 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.646591 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.646453 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg"] Nov 02 01:25:14.666316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.666284 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:14.685239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.684213 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.685239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.684875 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.685239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.684902 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.685239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.684965 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s82r\" (UniqueName: \"kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.729776 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.729737 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-b9d816-serving-cert\"" Nov 02 01:25:14.793124 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.793092 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.793354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.793176 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.793354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.793200 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.793354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.793258 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s82r\" (UniqueName: \"kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.794371 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.794344 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.798540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.798490 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.800779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.800753 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.809819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.809793 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s82r\" (UniqueName: \"kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r\") pod \"maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.907563 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.907534 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp"] Nov 02 01:25:14.908615 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:14.908590 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21d48af_7b46_4cf8_83f9_7d2e0d74d061.slice/crio-28c0558d522f7ac037bdf8a633b23baf5b7ef1651a01267befeb2f7f90f6c5d8 WatchSource:0}: Error finding container 28c0558d522f7ac037bdf8a633b23baf5b7ef1651a01267befeb2f7f90f6c5d8: Status 404 returned error can't find the container with id 28c0558d522f7ac037bdf8a633b23baf5b7ef1651a01267befeb2f7f90f6c5d8 Nov 02 01:25:14.954214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.954190 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:14.972891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:14.972868 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8"] Nov 02 01:25:14.974037 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:14.974006 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0187afc3_ac44_4424_b308_c332cdaac014.slice/crio-8b22f7cdb149d4e5a714398c577b43d36708a91a89294408c022ef9124f4c503 WatchSource:0}: Error finding container 8b22f7cdb149d4e5a714398c577b43d36708a91a89294408c022ef9124f4c503: Status 404 returned error can't find the container with id 8b22f7cdb149d4e5a714398c577b43d36708a91a89294408c022ef9124f4c503 Nov 02 01:25:15.076351 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.076315 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" event={"ID":"f21d48af-7b46-4cf8-83f9-7d2e0d74d061","Type":"ContainerStarted","Data":"5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090"} Nov 02 01:25:15.076495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.076379 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:15.076495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.076406 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" event={"ID":"f21d48af-7b46-4cf8-83f9-7d2e0d74d061","Type":"ContainerStarted","Data":"28c0558d522f7ac037bdf8a633b23baf5b7ef1651a01267befeb2f7f90f6c5d8"} Nov 02 01:25:15.077961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.077937 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" event={"ID":"0187afc3-ac44-4424-b308-c332cdaac014","Type":"ContainerStarted","Data":"a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a"} Nov 02 01:25:15.078080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.077980 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" event={"ID":"0187afc3-ac44-4424-b308-c332cdaac014","Type":"ContainerStarted","Data":"8b22f7cdb149d4e5a714398c577b43d36708a91a89294408c022ef9124f4c503"} Nov 02 01:25:15.078080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.078004 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:15.079466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.079441 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" event={"ID":"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4","Type":"ContainerStarted","Data":"20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73"} Nov 02 01:25:15.079574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.079484 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" event={"ID":"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4","Type":"ContainerStarted","Data":"60ef273828f4cc84917a5b3c4484d332ecefaa9e562193a4587a0e198c79ddd1"} Nov 02 01:25:15.079574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.079515 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:15.080889 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.080856 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" event={"ID":"1b58629a-2db8-4e59-a467-f9b597164194","Type":"ContainerStarted","Data":"edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e"} Nov 02 01:25:15.080969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.080903 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" event={"ID":"1b58629a-2db8-4e59-a467-f9b597164194","Type":"ContainerStarted","Data":"a1b2ae250ca1a26b860a4e2c044ef3aaa6776f5150681ed11c7ccf7a54aefa5c"} Nov 02 01:25:15.080969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.080923 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:15.099763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.099722 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" podStartSLOduration=1.099709724 podStartE2EDuration="1.099709724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:15.097508135 +0000 UTC m=+615.203297051" watchObservedRunningTime="2026-11-02 01:25:15.099709724 +0000 UTC m=+615.205498634" Nov 02 01:25:15.131006 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.130962 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" podStartSLOduration=1.13094923 podStartE2EDuration="1.13094923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:15.124500115 +0000 UTC m=+615.230289029" watchObservedRunningTime="2026-11-02 01:25:15.13094923 +0000 UTC m=+615.236738190" Nov 02 01:25:15.146077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.146034 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" podStartSLOduration=2.146020058 podStartE2EDuration="2.146020058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:15.143628108 +0000 UTC m=+615.249417024" watchObservedRunningTime="2026-11-02 01:25:15.146020058 +0000 UTC m=+615.251809006" Nov 02 01:25:15.167425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.167377 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" podStartSLOduration=2.167363554 podStartE2EDuration="2.167363554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:15.164225508 +0000 UTC m=+615.270014422" watchObservedRunningTime="2026-11-02 01:25:15.167363554 +0000 UTC m=+615.273152467" Nov 02 01:25:15.226184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.226154 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj"] Nov 02 01:25:15.230386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.230361 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.242787 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.242581 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj"] Nov 02 01:25:15.250324 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.250299 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg"] Nov 02 01:25:15.252614 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:15.252575 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0aa9a_70cd_434b_b2a9_a502ff2ae86d.slice/crio-f979dac47d27e7c7741dfe6777d601fd74d4bd01c51e370e8cd88cf9e10d70b7 WatchSource:0}: Error finding container f979dac47d27e7c7741dfe6777d601fd74d4bd01c51e370e8cd88cf9e10d70b7: Status 404 returned error can't find the container with id f979dac47d27e7c7741dfe6777d601fd74d4bd01c51e370e8cd88cf9e10d70b7 Nov 02 01:25:15.254774 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.254754 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:25:15.273564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.273541 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-123986\"" Nov 02 01:25:15.399646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.399612 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.399814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.399743 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4fd\" (UniqueName: \"kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd\") pod \"payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.500909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.500871 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.501254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.500979 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4fd\" (UniqueName: \"kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd\") pod \"payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.502048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.502022 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.516090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.516024 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4fd\" (UniqueName: \"kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd\") pod \"payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.555259 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.555231 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:15.630209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.628019 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5"] Nov 02 01:25:15.633975 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.633956 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.643028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.641912 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5"] Nov 02 01:25:15.643028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.642420 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5687bc-serving-cert\"" Nov 02 01:25:15.804311 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.804225 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.804476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.804387 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59msr\" (UniqueName: \"kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.804532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.804497 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.804582 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.804560 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.852293 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:15.850378 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8506fe_c38f_4ed3_aaad_e7f86fc64c6c.slice/crio-19c7c36c851280899e0a9df5c2f1e14fb0084999c7739fdc614da509eb7606db WatchSource:0}: Error finding container 19c7c36c851280899e0a9df5c2f1e14fb0084999c7739fdc614da509eb7606db: Status 404 returned error can't find the container with id 19c7c36c851280899e0a9df5c2f1e14fb0084999c7739fdc614da509eb7606db Nov 02 01:25:15.852293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.851471 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj"] Nov 02 01:25:15.906332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.906307 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59msr\" (UniqueName: \"kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.906428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.906372 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.906428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.906408 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.906558 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.906542 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.907180 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.907152 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.909038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.909016 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.909147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.909049 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.917829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.917808 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59msr\" (UniqueName: \"kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr\") pod \"maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:15.958704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:15.958671 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:16.094148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.094073 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" event={"ID":"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d","Type":"ContainerStarted","Data":"2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4"} Nov 02 01:25:16.094296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.094182 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" event={"ID":"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d","Type":"ContainerStarted","Data":"f979dac47d27e7c7741dfe6777d601fd74d4bd01c51e370e8cd88cf9e10d70b7"} Nov 02 01:25:16.094296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.094250 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:16.100798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.099279 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" event={"ID":"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c","Type":"ContainerStarted","Data":"182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba"} Nov 02 01:25:16.100798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.099347 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" event={"ID":"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c","Type":"ContainerStarted","Data":"19c7c36c851280899e0a9df5c2f1e14fb0084999c7739fdc614da509eb7606db"} Nov 02 01:25:16.100798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.099819 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:16.119821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.119777 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" podStartSLOduration=1.395219488 podStartE2EDuration="2.119750111s" totalImagesPullingTime="724.530623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.118279776 +0000 UTC m=+616.224068690" watchObservedRunningTime="2026-11-02 01:25:16.119750111 +0000 UTC m=+616.225539058" Nov 02 01:25:16.143746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.143360 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" podStartSLOduration=1.143342928 podStartE2EDuration="1.143342928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.142687274 +0000 UTC m=+616.248476190" watchObservedRunningTime="2026-11-02 01:25:16.143342928 +0000 UTC m=+616.249131845" Nov 02 01:25:16.277313 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:16.277276 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642275e4_de7e_4784_92a6_05f1bc69095b.slice/crio-64e6d8eebd0727728d5416d730ab3b2123e469b4ede74960df42d876a01977ff WatchSource:0}: Error finding container 64e6d8eebd0727728d5416d730ab3b2123e469b4ede74960df42d876a01977ff: Status 404 returned error can't find the container with id 64e6d8eebd0727728d5416d730ab3b2123e469b4ede74960df42d876a01977ff Nov 02 01:25:16.277605 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:16.277581 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5"] Nov 02 01:25:17.110403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.110296 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" event={"ID":"642275e4-de7e-4784-92a6-05f1bc69095b","Type":"ContainerStarted","Data":"492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04"} Nov 02 01:25:17.110403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.110359 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" event={"ID":"642275e4-de7e-4784-92a6-05f1bc69095b","Type":"ContainerStarted","Data":"64e6d8eebd0727728d5416d730ab3b2123e469b4ede74960df42d876a01977ff"} Nov 02 01:25:17.132913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.132859 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" podStartSLOduration=1.5931606870000001 podStartE2EDuration="2.132843282s" totalImagesPullingTime="539.682595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:17.129347094 +0000 UTC m=+617.235136009" watchObservedRunningTime="2026-11-02 01:25:17.132843282 +0000 UTC m=+617.238632203" Nov 02 01:25:17.326856 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.326817 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc"] Nov 02 01:25:17.331183 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.331154 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.344319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.344293 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc"] Nov 02 01:25:17.345350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.345328 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5ed42b\"" Nov 02 01:25:17.425254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.425170 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.425404 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.425359 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2w5\" (UniqueName: \"kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5\") pod \"payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.526209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.526178 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2w5\" (UniqueName: \"kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5\") pod \"payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.526394 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.526231 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.526922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.526903 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.540595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.540561 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2w5\" (UniqueName: \"kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5\") pod \"payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.551760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.551730 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk"] Nov 02 01:25:17.556943 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.556916 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.562436 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.562414 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-47dd19-serving-cert\"" Nov 02 01:25:17.580009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.579979 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk"] Nov 02 01:25:17.626822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.626792 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68k54\" (UniqueName: \"kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.626968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.626842 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.626968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.626948 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.627052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.626975 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.651692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.651665 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:17.727809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.727781 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.727932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.727817 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.727932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.727916 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68k54\" (UniqueName: \"kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.728022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.727954 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.728637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.728597 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.730419 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.730398 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.730528 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.730475 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.740052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.740027 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68k54\" (UniqueName: \"kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54\") pod \"maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.882056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.882020 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:17.934513 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.934485 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc"] Nov 02 01:25:17.935714 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:17.935679 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efc4e7e_a049_4397_9fcf_2eae203cb81b.slice/crio-62d56f0f6bc37b2057f0977bba75fe1f48fa3494f379acbe00aa67c14d7b6625 WatchSource:0}: Error finding container 62d56f0f6bc37b2057f0977bba75fe1f48fa3494f379acbe00aa67c14d7b6625: Status 404 returned error can't find the container with id 62d56f0f6bc37b2057f0977bba75fe1f48fa3494f379acbe00aa67c14d7b6625 Nov 02 01:25:17.974744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:17.974718 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:25:18.120793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:18.120745 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" event={"ID":"4efc4e7e-a049-4397-9fcf-2eae203cb81b","Type":"ContainerStarted","Data":"bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887"} Nov 02 01:25:18.121299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:18.120818 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" event={"ID":"4efc4e7e-a049-4397-9fcf-2eae203cb81b","Type":"ContainerStarted","Data":"62d56f0f6bc37b2057f0977bba75fe1f48fa3494f379acbe00aa67c14d7b6625"} Nov 02 01:25:18.121299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:18.121286 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:18.142172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:18.142105 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" podStartSLOduration=1.142093525 podStartE2EDuration="1.142093525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:18.140871693 +0000 UTC m=+618.246660610" watchObservedRunningTime="2026-11-02 01:25:18.142093525 +0000 UTC m=+618.247882484" Nov 02 01:25:18.183237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:18.183211 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk"] Nov 02 01:25:18.186246 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:18.186212 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de0b2fd_ecb6_4f9b_91d4_bd4542f077e1.slice/crio-099dfbdb616a4d50cfeb69de5285a3e56d054c21258f8b05a78c20a1163448b1 WatchSource:0}: Error finding container 099dfbdb616a4d50cfeb69de5285a3e56d054c21258f8b05a78c20a1163448b1: Status 404 returned error can't find the container with id 099dfbdb616a4d50cfeb69de5285a3e56d054c21258f8b05a78c20a1163448b1 Nov 02 01:25:19.129919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:19.129761 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" event={"ID":"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1","Type":"ContainerStarted","Data":"6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081"} Nov 02 01:25:19.129919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:19.129825 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" event={"ID":"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1","Type":"ContainerStarted","Data":"099dfbdb616a4d50cfeb69de5285a3e56d054c21258f8b05a78c20a1163448b1"} Nov 02 01:25:19.130409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:19.129921 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:19.151700 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:19.151653 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" podStartSLOduration=1.46385575 podStartE2EDuration="2.151639869s" totalImagesPullingTime="687.784119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:19.148546631 +0000 UTC m=+619.254335545" watchObservedRunningTime="2026-11-02 01:25:19.151639869 +0000 UTC m=+619.257428783" Nov 02 01:25:20.136473 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:20.136440 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:21.428839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.428784 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h"] Nov 02 01:25:21.433840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.433814 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.448534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.446791 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5ed42b-serving-cert\"" Nov 02 01:25:21.456302 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.456278 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h"] Nov 02 01:25:21.464949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.464917 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.465081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.464997 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm42x\" (UniqueName: \"kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.465206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.465180 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.465340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.465322 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.567307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.567231 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm42x\" (UniqueName: \"kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.567661 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.567631 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.567804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.567751 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.567876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.567833 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.568788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.568673 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.571873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.571848 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.574154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.573461 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.579723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.579698 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm42x\" (UniqueName: \"kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x\") pod \"maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:21.760727 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:21.760690 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:22.054485 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.054458 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h"] Nov 02 01:25:22.055549 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:22.055506 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6322f4_7380_4f9a_bd41_6fa8dfd98bff.slice/crio-3bafa34860a94965ef00c71c7482778211bc8861685b693e97c926641a3c6825 WatchSource:0}: Error finding container 3bafa34860a94965ef00c71c7482778211bc8861685b693e97c926641a3c6825: Status 404 returned error can't find the container with id 3bafa34860a94965ef00c71c7482778211bc8861685b693e97c926641a3c6825 Nov 02 01:25:22.117763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.117737 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:25:22.138335 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.138301 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:25:22.154106 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.154069 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" event={"ID":"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff","Type":"ContainerStarted","Data":"3bafa34860a94965ef00c71c7482778211bc8861685b693e97c926641a3c6825"} Nov 02 01:25:22.260792 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.260587 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6"] Nov 02 01:25:22.268196 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.268172 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.290997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.290557 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b6ec07-serving-cert\"" Nov 02 01:25:22.295081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.295051 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6"] Nov 02 01:25:22.387173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.387077 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.387173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.387146 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.387353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.387177 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.387353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.387258 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjq6\" (UniqueName: \"kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.488662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.488632 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.489191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.488699 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjq6\" (UniqueName: \"kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.489191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.488892 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.489191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.488919 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.489631 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.489607 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.491326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.491301 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.491412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.491335 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.504210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.504190 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjq6\" (UniqueName: \"kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6\") pod \"maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.601164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.601128 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:22.906483 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:22.906446 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6"] Nov 02 01:25:22.907631 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:25:22.907599 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdc3bde_7cc5_4058_9f65_92000449176a.slice/crio-056b6f95f8b9232651b10a1b3f25bd2673bf67f09be53bbeb6f8a7462830136a WatchSource:0}: Error finding container 056b6f95f8b9232651b10a1b3f25bd2673bf67f09be53bbeb6f8a7462830136a: Status 404 returned error can't find the container with id 056b6f95f8b9232651b10a1b3f25bd2673bf67f09be53bbeb6f8a7462830136a Nov 02 01:25:23.165261 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:23.165219 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" event={"ID":"2fdc3bde-7cc5-4058-9f65-92000449176a","Type":"ContainerStarted","Data":"056b6f95f8b9232651b10a1b3f25bd2673bf67f09be53bbeb6f8a7462830136a"} Nov 02 01:25:23.167146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:23.167089 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" event={"ID":"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff","Type":"ContainerStarted","Data":"143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6"} Nov 02 01:25:23.167289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:23.167181 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:23.188265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:23.188219 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" podStartSLOduration=1.649947488 podStartE2EDuration="2.188206516s" totalImagesPullingTime="538.259028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:23.185991091 +0000 UTC m=+623.291780006" watchObservedRunningTime="2026-11-02 01:25:23.188206516 +0000 UTC m=+623.293995430" Nov 02 01:25:24.135294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:24.135265 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:25:24.179001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:24.178962 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" event={"ID":"2fdc3bde-7cc5-4058-9f65-92000449176a","Type":"ContainerStarted","Data":"de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9"} Nov 02 01:25:24.179254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:24.179234 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:24.201818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:24.201756 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" podStartSLOduration=1.465322162 podStartE2EDuration="2.201736475s" totalImagesPullingTime="736.414313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:24.198530753 +0000 UTC m=+624.304319668" watchObservedRunningTime="2026-11-02 01:25:24.201736475 +0000 UTC m=+624.307525391" Nov 02 01:25:26.102826 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:26.102796 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:25:26.103350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:26.103068 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:25:26.103442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:26.103423 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:25:26.103496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:26.103479 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:25:26.151287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:26.151254 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:25:26.616259 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:26.616224 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:25:27.752058 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:25:27.752017 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:25:28.124696 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:28.124615 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:25:29.184788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:29.184759 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:25:30.140354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:30.140330 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:25:30.191815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:30.191763 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:25:33.672574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:33.672536 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:25:42.912801 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:42.912752 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:25:54.250385 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:25:54.250354 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:25:57.751240 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:25:57.751205 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:26:09.438582 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.438489 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:26:09.790984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.790945 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5"] Nov 02 01:26:09.797467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.797443 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:09.805461 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.804539 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"default-dockercfg-sflcv\"" Nov 02 01:26:09.809076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.808271 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"premium-w3-5ed42b-kserve-self-signed-certs\"" Nov 02 01:26:09.809076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.808508 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"kube-root-ca.crt\"" Nov 02 01:26:09.809076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.808613 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"openshift-service-ca.crt\"" Nov 02 01:26:09.809076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.808735 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5"] Nov 02 01:26:09.951234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.951191 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:09.951439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.951244 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:09.951439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.951283 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:09.951439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.951408 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52swk\" (UniqueName: \"kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:09.951613 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.951444 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:09.951668 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:09.951630 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.052630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.052541 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.052630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.052623 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.052863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.052639 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.052863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.052660 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.052863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.052686 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52swk\" (UniqueName: \"kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.052863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.052713 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.053085 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.053046 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.053174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.053077 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.053174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.053104 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.055004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.054973 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.055701 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.055671 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.069373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.069349 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52swk\" (UniqueName: \"kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk\") pod \"premium-w3-5ed42b-kserve-5dc6d8d885-484x5\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.114536 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.114506 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:10.419616 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.419593 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5"] Nov 02 01:26:10.422036 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:26:10.422009 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ba69e99_2e61_4fbd_960d_a5dd51e67ab6.slice/crio-77ff51209a60c23563fff857708b7e88d747d38db2dc8ac12588d5c5d493f86e WatchSource:0}: Error finding container 77ff51209a60c23563fff857708b7e88d747d38db2dc8ac12588d5c5d493f86e: Status 404 returned error can't find the container with id 77ff51209a60c23563fff857708b7e88d747d38db2dc8ac12588d5c5d493f86e Nov 02 01:26:10.539875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.539845 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" event={"ID":"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6","Type":"ContainerStarted","Data":"8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0"} Nov 02 01:26:10.540273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.539883 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" event={"ID":"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6","Type":"ContainerStarted","Data":"77ff51209a60c23563fff857708b7e88d747d38db2dc8ac12588d5c5d493f86e"} Nov 02 01:26:10.561753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:10.561708 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podStartSLOduration=1.561684879 podStartE2EDuration="1.561684879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:10.558487729 +0000 UTC m=+670.664276643" watchObservedRunningTime="2026-11-02 01:26:10.561684879 +0000 UTC m=+670.667473796" Nov 02 01:26:11.115036 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:11.115003 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:20.115653 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:20.115598 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:20.128950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:20.128924 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:20.623048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:20.623014 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:26:27.754416 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:26:27.750903 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:26:34.753333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:26:34.753284 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:26:57.751982 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:26:57.751944 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:27:10.361854 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:10.361819 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:27:17.636256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.636219 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l"] Nov 02 01:27:17.641033 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.641008 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.648047 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.648020 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l"] Nov 02 01:27:17.651154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.650921 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"distinct-w3-5ed42b-kserve-self-signed-certs\"" Nov 02 01:27:17.694760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.694709 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.694951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.694783 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.694951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.694903 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.695055 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.694979 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.695109 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.695088 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.695212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.695196 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.795862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.795818 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.795915 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796020 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796341 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796502 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796542 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796654 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.796704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796659 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.797017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.796998 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.799602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.799556 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.799710 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.799622 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.808240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.808217 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x\") pod \"distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:17.958666 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:17.958566 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:18.241164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:18.241134 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l"] Nov 02 01:27:18.242865 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:27:18.242833 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod997f84c9_3953_470f_bb33_ba1f0375ddd2.slice/crio-c8ef752d252979f731e394e03019cca32bb99acd6e2ce7801a192c17fb6d5f0b WatchSource:0}: Error finding container c8ef752d252979f731e394e03019cca32bb99acd6e2ce7801a192c17fb6d5f0b: Status 404 returned error can't find the container with id c8ef752d252979f731e394e03019cca32bb99acd6e2ce7801a192c17fb6d5f0b Nov 02 01:27:18.356707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:18.356660 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" event={"ID":"997f84c9-3953-470f-bb33-ba1f0375ddd2","Type":"ContainerStarted","Data":"8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10"} Nov 02 01:27:18.356878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:18.356716 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" event={"ID":"997f84c9-3953-470f-bb33-ba1f0375ddd2","Type":"ContainerStarted","Data":"c8ef752d252979f731e394e03019cca32bb99acd6e2ce7801a192c17fb6d5f0b"} Nov 02 01:27:18.378892 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:18.378833 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podStartSLOduration=1.378818595 podStartE2EDuration="1.378818595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:18.375441575 +0000 UTC m=+738.481230502" watchObservedRunningTime="2026-11-02 01:27:18.378818595 +0000 UTC m=+738.484607515" Nov 02 01:27:18.959456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:18.959419 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:24.028347 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.028305 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd"] Nov 02 01:27:24.032750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.032726 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.048379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.048072 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47dd19-openshift-default-dockercfg-tx9jn\"" Nov 02 01:27:24.054492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.054466 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd"] Nov 02 01:27:24.168237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168201 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168249 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168326 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168438 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168554 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nr4\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168608 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168683 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168716 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.168783 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.168747 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.238246 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.238195 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7"] Nov 02 01:27:24.250879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.250827 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.256087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.255637 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-123986-openshift-default-dockercfg-8gxrz\"" Nov 02 01:27:24.256281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.256176 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7"] Nov 02 01:27:24.270410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270375 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270570 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270427 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270570 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270456 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270570 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270551 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270734 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270589 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270734 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270615 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270734 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270665 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270755 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nr4\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.270885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.270788 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.271181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.271154 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.271181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.271430 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.271181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.271674 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.273633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.273608 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.274383 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.273921 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.274500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.274073 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.277365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.276463 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.284410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.284347 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.290921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.290892 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nr4\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.348755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.348727 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:24.371888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.371854 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372024 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.371968 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372024 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372009 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372087 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372157 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372223 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372296 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372396 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.372466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.372449 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjc25\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474352 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474309 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474541 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474386 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474541 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474418 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474541 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474449 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474541 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474476 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474541 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474612 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474719 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474767 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjc25\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.474965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.474920 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.475241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.475216 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.475551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.475509 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.475551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.475544 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.475821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.475798 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.477753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.477693 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.477753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.477739 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.486712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.486687 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.487238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.487212 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjc25\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.576097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.576013 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:24.646187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.645798 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:24.646187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.645893 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:24.646187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.645935 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:24.674200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.667658 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd"] Nov 02 01:27:24.867836 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:27:24.867796 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e0813c_0557_4448_9cd2_ee5c591cae4a.slice/crio-19b3f1cbf0ccc4d43395d5522cbe1a6d518110a1aa5eb970339e0fc2e4cb2845 WatchSource:0}: Error finding container 19b3f1cbf0ccc4d43395d5522cbe1a6d518110a1aa5eb970339e0fc2e4cb2845: Status 404 returned error can't find the container with id 19b3f1cbf0ccc4d43395d5522cbe1a6d518110a1aa5eb970339e0fc2e4cb2845 Nov 02 01:27:24.869781 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.869755 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7"] Nov 02 01:27:24.870576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.870537 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:24.870673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.870608 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:24.870673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:24.870642 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:25.410134 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.410085 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" event={"ID":"d8e0813c-0557-4448-9cd2-ee5c591cae4a","Type":"ContainerStarted","Data":"8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771"} Nov 02 01:27:25.410597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.410178 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" event={"ID":"d8e0813c-0557-4448-9cd2-ee5c591cae4a","Type":"ContainerStarted","Data":"19b3f1cbf0ccc4d43395d5522cbe1a6d518110a1aa5eb970339e0fc2e4cb2845"} Nov 02 01:27:25.411935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.411911 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" event={"ID":"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae","Type":"ContainerStarted","Data":"194bc8dc5bb761a7f03a33d633c4f6cc025f5729bb9c205231e019d3982b02ea"} Nov 02 01:27:25.412044 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.411959 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" event={"ID":"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae","Type":"ContainerStarted","Data":"5674ad33d3004e86cff78525b509760484d2feaab77cb8047a7390c7b572056a"} Nov 02 01:27:25.451144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.445405 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podStartSLOduration=1.445383718 podStartE2EDuration="1.445383718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:25.432004356 +0000 UTC m=+745.537793272" watchObservedRunningTime="2026-11-02 01:27:25.445383718 +0000 UTC m=+745.551172633" Nov 02 01:27:25.484139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.484028 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podStartSLOduration=1.48400964 podStartE2EDuration="1.48400964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:25.482909587 +0000 UTC m=+745.588698498" watchObservedRunningTime="2026-11-02 01:27:25.48400964 +0000 UTC m=+745.589798562" Nov 02 01:27:25.576518 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.576474 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:25.578505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.578480 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:25.578616 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:25.578528 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:26.353142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.350888 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:26.358141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.354070 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:27:26.358141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.354160 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 01:27:26.580145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.578536 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:26.580145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.578613 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:26.824191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.824079 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d"] Nov 02 01:27:26.829226 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.828976 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d"] Nov 02 01:27:26.829226 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.829145 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:26.833632 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.833604 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"openshift-service-ca.crt\"" Nov 02 01:27:26.833771 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.833714 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"distinct-w6-b9d816-kserve-self-signed-certs\"" Nov 02 01:27:26.833825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.833780 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"kube-root-ca.crt\"" Nov 02 01:27:26.833990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.833957 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"default-dockercfg-7z7ft\"" Nov 02 01:27:26.926178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.925403 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:26.926178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.925466 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:26.926178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.925614 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:26.926178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.925647 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:26.926178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.925721 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78pv\" (UniqueName: \"kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:26.926178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:26.925749 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.028728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.028683 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.028728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.028733 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.028991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.028790 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78pv\" (UniqueName: \"kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.028991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.028815 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.028991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.028868 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.028991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.028906 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.029494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.029470 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.029602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.029548 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.029735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.029717 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.032134 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.032080 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.032262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.032245 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.058178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.044544 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78pv\" (UniqueName: \"kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv\") pod \"distinct-w6-b9d816-kserve-67f8c9d989-whh8d\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.211200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.210352 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:27.412241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.411193 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:27:27.412241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.411632 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 01:27:27.606178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.606061 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:27.606178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.606148 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:27.707161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.703189 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d"] Nov 02 01:27:27.814152 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:27:27.813901 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:27:27.966142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:27.963265 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:28.012199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.012173 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:28.357834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.357797 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:27:28.358143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.358094 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 01:27:28.452207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.452172 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" event={"ID":"deb78ec3-fde7-42fd-ae62-4d03e15f2190","Type":"ContainerStarted","Data":"bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845"} Nov 02 01:27:28.452439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.452420 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" event={"ID":"deb78ec3-fde7-42fd-ae62-4d03e15f2190","Type":"ContainerStarted","Data":"9210cf2b9fa39739915d70951097c16f3c00321297ebe847507b2e7e6c18e273"} Nov 02 01:27:28.475772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.475708 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podStartSLOduration=2.475688801 podStartE2EDuration="2.475688801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:28.475497013 +0000 UTC m=+748.581285947" watchObservedRunningTime="2026-11-02 01:27:28.475688801 +0000 UTC m=+748.581477716" Nov 02 01:27:28.513591 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.513559 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:27:28.576543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.576507 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:28.576814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:28.576790 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:29.210893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:29.210861 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:29.356144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:29.352527 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:27:29.356144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:29.352593 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 01:27:29.576559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:29.576517 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:29.577852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:29.577817 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:30.349618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:30.349573 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:27:30.350304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:30.350268 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 01:27:30.585141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:30.584539 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:30.585141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:30.584605 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:31.349381 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:31.349339 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:27:31.349574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:31.349411 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 01:27:31.577177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:31.577110 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 01:27:31.577683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:31.577198 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 01:27:32.580797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:32.580759 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:32.581265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:32.581156 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:32.582009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:32.581987 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:27:32.607600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:32.607568 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:32.608047 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:32.608019 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:32.609157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:32.609134 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:27:37.211379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:37.211328 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:37.224851 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:37.224827 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:37.582356 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:37.582317 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:27:38.812547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.812515 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn"] Nov 02 01:27:38.816938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.816917 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.823692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.823646 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b9d816-openshift-default-dockercfg-hvdvt\"" Nov 02 01:27:38.831063 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.831033 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn"] Nov 02 01:27:38.873970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.873943 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874107 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.873974 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874107 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874083 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874142 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874165 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874247 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874289 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874321 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.874466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.874445 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvzfq\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.975713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975620 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.975713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975685 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.975945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975724 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzfq\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.975945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975775 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.975945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975800 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.975945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975906 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975955 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.975993 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.976052 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.976106 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.976136 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976499 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.976270 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.976521 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.976821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.976797 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.978559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.978522 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.978732 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.978716 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.988094 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.988068 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:38.988421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:38.988401 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzfq\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:39.052969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.052928 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf"] Nov 02 01:27:39.059654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.059622 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.064883 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.064819 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b6ec07-openshift-default-dockercfg-9l7xh\"" Nov 02 01:27:39.077163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.076827 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v274v\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.076875 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.076989 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.077026 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.077100 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.077151 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.077175 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.077257 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.077552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.077294 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.088383 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.088352 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf"] Nov 02 01:27:39.132290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.132256 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:39.179605 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179569 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179624 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179680 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179710 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179734 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179806 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179844 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179932 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v274v\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.179989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.179967 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.180503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.180479 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.180954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.180929 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.181132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.181099 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.181386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.181359 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.181577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.181558 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.184107 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.184084 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.184909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.184885 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.193694 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.193669 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.196085 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.196062 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v274v\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.389684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.389604 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:39.444937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.444799 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn"] Nov 02 01:27:39.444937 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:27:39.444898 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3943a20_f352_4b03_bb3b_402d97c5e103.slice/crio-1ad15dbf51855383773b6795d10561bea0ac75f253cca380325bc23bd284aea9 WatchSource:0}: Error finding container 1ad15dbf51855383773b6795d10561bea0ac75f253cca380325bc23bd284aea9: Status 404 returned error can't find the container with id 1ad15dbf51855383773b6795d10561bea0ac75f253cca380325bc23bd284aea9 Nov 02 01:27:39.448383 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.448345 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:39.448544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.448432 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:39.448544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.448483 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:39.586088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.586051 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" event={"ID":"a3943a20-f352-4b03-bb3b-402d97c5e103","Type":"ContainerStarted","Data":"4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990"} Nov 02 01:27:39.586273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.586131 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" event={"ID":"a3943a20-f352-4b03-bb3b-402d97c5e103","Type":"ContainerStarted","Data":"1ad15dbf51855383773b6795d10561bea0ac75f253cca380325bc23bd284aea9"} Nov 02 01:27:39.614371 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.614327 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podStartSLOduration=1.614311806 podStartE2EDuration="1.614311806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:39.611152469 +0000 UTC m=+759.716941396" watchObservedRunningTime="2026-11-02 01:27:39.614311806 +0000 UTC m=+759.720100719" Nov 02 01:27:39.696067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.696044 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf"] Nov 02 01:27:39.696546 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:27:39.696525 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e0dd6f_2ed0_4d94_9d4d_a40952a0ea2d.slice/crio-254294ccddd83eee13b71ebabbe386f8476ac8480208dcda533daaaa79442c63 WatchSource:0}: Error finding container 254294ccddd83eee13b71ebabbe386f8476ac8480208dcda533daaaa79442c63: Status 404 returned error can't find the container with id 254294ccddd83eee13b71ebabbe386f8476ac8480208dcda533daaaa79442c63 Nov 02 01:27:39.699163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.699111 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:39.699244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.699194 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:39.699244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:39.699222 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:40.132492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.132458 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:40.133973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.133941 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:40.134078 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.133992 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:40.241143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.237091 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:27:40.609457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.609413 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" event={"ID":"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d","Type":"ContainerStarted","Data":"8be8a5df4abbe35b624d5269026f45a9172eaf163a5cea4b98755dd842d98b5f"} Nov 02 01:27:40.609645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.609476 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" event={"ID":"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d","Type":"ContainerStarted","Data":"254294ccddd83eee13b71ebabbe386f8476ac8480208dcda533daaaa79442c63"} Nov 02 01:27:40.634959 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:40.634912 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podStartSLOduration=1.634899887 podStartE2EDuration="1.634899887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:40.631525922 +0000 UTC m=+760.737314837" watchObservedRunningTime="2026-11-02 01:27:40.634899887 +0000 UTC m=+760.740688801" Nov 02 01:27:41.133631 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:41.133594 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:41.134035 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:41.133647 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:41.397233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:41.395229 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:41.397233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:41.395389 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:27:41.397233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:41.395438 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 01:27:42.162576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:42.162538 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:42.164238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:42.164190 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:42.392951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:42.392489 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:27:42.392951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:42.392567 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 01:27:43.144697 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:43.144661 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:43.144993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:43.144963 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:43.398201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:43.393626 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:27:43.398201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:43.393687 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 01:27:44.133271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:44.133239 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:44.133473 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:44.133291 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:44.396201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:44.395571 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:27:44.396201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:44.395634 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 01:27:45.132730 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:45.132658 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:45.132730 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:45.132715 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:45.391003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:45.390498 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:27:45.391003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:45.390562 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 01:27:46.133733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:46.133683 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:27:46.134256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:46.133755 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 01:27:46.390429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:46.390342 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:27:46.390429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:46.390409 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 01:27:47.706666 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:47.706630 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:47.707317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:47.707011 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:47.708450 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:47.708424 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:27:47.905861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:47.905830 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:47.906341 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:47.906317 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:47.908326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:47.908298 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:27:54.312603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.312534 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h"] Nov 02 01:27:54.318902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.318875 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.335825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.335799 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h"] Nov 02 01:27:54.337350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.337329 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5ed42b-openshift-default-dockercfg-6p624\"" Nov 02 01:27:54.348916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.348892 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2"] Nov 02 01:27:54.354494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.354472 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.370910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.370827 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.383910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.383879 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.384170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.384153 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.384353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.384334 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.384504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.384488 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.384765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.384749 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.384897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.384883 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.385041 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.385027 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcq9g\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.385226 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.385209 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.402226 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.402189 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2"] Nov 02 01:27:54.435183 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.435154 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ca1c53-openshift-default-dockercfg-hhbph\"" Nov 02 01:27:54.486717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486687 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.486923 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486726 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.486923 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486769 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.486923 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486801 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcq9g\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.486923 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486837 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.486923 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486866 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.487215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.486952 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.487215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487025 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcc6p\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.487215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487089 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487140 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487193 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487235 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487260 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487309 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.487472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487355 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487372 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487488 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.487731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487523 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.487731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487593 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487642 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487680 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.487731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.487711 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.488624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.488597 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.489905 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.489879 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.490034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.489999 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.499321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.499286 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.499895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.499869 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcq9g\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.589369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589292 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcc6p\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589402 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589445 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589469 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589535 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589641 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589690 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589729 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589755 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.589976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.589835 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.590036 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.590021 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.590097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.590070 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.590306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.590284 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.590554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.590532 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.591971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.591950 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.592207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.592188 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.601624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.601605 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcc6p\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.601724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.601636 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.640454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.640421 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:54.686369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.686331 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:54.933860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.933827 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h"] Nov 02 01:27:54.934921 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:27:54.934897 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773a918a_ee3a_4991_bc33_5ac626a65237.slice/crio-2323697036d1e23dd594e6a3917be5fbaf778881b9508dd29441d8022a402e77 WatchSource:0}: Error finding container 2323697036d1e23dd594e6a3917be5fbaf778881b9508dd29441d8022a402e77: Status 404 returned error can't find the container with id 2323697036d1e23dd594e6a3917be5fbaf778881b9508dd29441d8022a402e77 Nov 02 01:27:54.937440 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.937402 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:54.937527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.937468 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:54.937527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.937499 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:54.996448 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:54.996415 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2"] Nov 02 01:27:55.006256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.006207 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:55.006452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.006362 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:55.006452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.006459 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:27:55.812266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.812224 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" event={"ID":"773a918a-ee3a-4991-bc33-5ac626a65237","Type":"ContainerStarted","Data":"47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107"} Nov 02 01:27:55.812697 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.812632 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" event={"ID":"773a918a-ee3a-4991-bc33-5ac626a65237","Type":"ContainerStarted","Data":"2323697036d1e23dd594e6a3917be5fbaf778881b9508dd29441d8022a402e77"} Nov 02 01:27:55.814094 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.814068 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" event={"ID":"37a89abf-f2a4-4fcc-8129-337130c09ad5","Type":"ContainerStarted","Data":"aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f"} Nov 02 01:27:55.814238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.814107 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" event={"ID":"37a89abf-f2a4-4fcc-8129-337130c09ad5","Type":"ContainerStarted","Data":"cada15b5bd8f006da0f8a5ca367a9ed43629647dc1b7e3d0960cd32b9fc83a55"} Nov 02 01:27:55.838185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:55.838110 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podStartSLOduration=1.838089923 podStartE2EDuration="1.838089923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:55.835362329 +0000 UTC m=+775.941151244" watchObservedRunningTime="2026-11-02 01:27:55.838089923 +0000 UTC m=+775.943878837" Nov 02 01:27:56.647274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:56.646817 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:27:56.647274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:56.646946 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 01:27:56.647274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:56.646991 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 01:27:56.705314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:56.705274 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:27:56.707364 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:56.707337 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 01:27:56.707507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:56.707391 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 01:27:57.641487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:57.641447 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 01:27:57.642082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:57.642050 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 01:27:57.688145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:57.687495 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 01:27:57.688145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:57.687559 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 01:27:57.779527 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:27:57.779488 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:27:58.647231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:58.647198 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 01:27:58.647824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:58.647793 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 01:27:58.695143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:58.687530 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 01:27:58.695143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:58.687590 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 01:27:59.643188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:59.641492 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 01:27:59.643188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:59.641551 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 01:27:59.690143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:59.688701 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 01:27:59.690143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:27:59.688779 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 01:28:00.645144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:00.641462 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 01:28:00.645144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:00.641526 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 01:28:00.690143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:00.689580 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 01:28:00.690143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:00.689643 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 01:28:01.643502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:01.643463 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 01:28:01.643724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:01.643519 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 01:28:01.687280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:01.687246 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 01:28:01.687475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:01.687310 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 01:28:02.244682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.244603 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podStartSLOduration=8.24458337 podStartE2EDuration="8.24458337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:55.880971737 +0000 UTC m=+775.986760653" watchObservedRunningTime="2026-11-02 01:28:02.24458337 +0000 UTC m=+782.350372284" Nov 02 01:28:02.246160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.246134 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx"] Nov 02 01:28:02.255674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.255648 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.267755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.263880 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx"] Nov 02 01:28:02.272184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.272087 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tdsjr\"" Nov 02 01:28:02.272538 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.272518 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 01:28:02.272631 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.272569 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 01:28:02.272631 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.272597 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 01:28:02.424734 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.424691 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.424734 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.424742 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.425019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.424889 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.425019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.424965 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd923345-a8ad-45e5-8a8f-ab39df962b65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.425019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.425000 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.425229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.425074 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.425229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.425135 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqk4k\" (UniqueName: \"kubernetes.io/projected/dd923345-a8ad-45e5-8a8f-ab39df962b65-kube-api-access-zqk4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.526706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526611 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.526706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526653 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd923345-a8ad-45e5-8a8f-ab39df962b65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.526706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526691 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.527038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526759 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.527038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526802 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqk4k\" (UniqueName: \"kubernetes.io/projected/dd923345-a8ad-45e5-8a8f-ab39df962b65-kube-api-access-zqk4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.527038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526906 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.527038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.526959 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.527753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.527722 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.530172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.530142 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.530283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.530142 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.530347 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.530319 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd923345-a8ad-45e5-8a8f-ab39df962b65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.530565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.530542 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd923345-a8ad-45e5-8a8f-ab39df962b65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.541228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.541204 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd923345-a8ad-45e5-8a8f-ab39df962b65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.541323 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.541265 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqk4k\" (UniqueName: \"kubernetes.io/projected/dd923345-a8ad-45e5-8a8f-ab39df962b65-kube-api-access-zqk4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q47qx\" (UID: \"dd923345-a8ad-45e5-8a8f-ab39df962b65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.629837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.629798 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:02.694156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.694105 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:28:02.694745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.694726 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:28:02.695739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.695721 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:28:02.755091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.738278 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:28:02.755091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.739316 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:28:02.755091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.742615 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:28:02.937133 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:02.937085 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx"] Nov 02 01:28:02.939706 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:28:02.939670 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd923345_a8ad_45e5_8a8f_ab39df962b65.slice/crio-a25a32639d64c4f77ccd91ccd500107a0619464c67498b983ef1751839de3c45 WatchSource:0}: Error finding container a25a32639d64c4f77ccd91ccd500107a0619464c67498b983ef1751839de3c45: Status 404 returned error can't find the container with id a25a32639d64c4f77ccd91ccd500107a0619464c67498b983ef1751839de3c45 Nov 02 01:28:03.912835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:03.912791 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" event={"ID":"dd923345-a8ad-45e5-8a8f-ab39df962b65","Type":"ContainerStarted","Data":"a25a32639d64c4f77ccd91ccd500107a0619464c67498b983ef1751839de3c45"} Nov 02 01:28:08.809671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.809635 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:08.810008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.809698 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:08.919045 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.919010 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz"] Nov 02 01:28:08.923855 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.923833 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:08.930542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.930524 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5687bc-openshift-default-dockercfg-mw8fh\"" Nov 02 01:28:08.937057 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.935898 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz"] Nov 02 01:28:08.962158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.961599 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" event={"ID":"dd923345-a8ad-45e5-8a8f-ab39df962b65","Type":"ContainerStarted","Data":"d846c7bacc7febdae818e925f12724d34e6d3f2ba43deac0207112a879e26ee1"} Nov 02 01:28:08.962524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:08.962502 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:09.018823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.018787 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.018912 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcx4\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.018971 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.019050 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.019185 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.019260 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.019357 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.019407 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.019477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.019436 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121379 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121426 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121486 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121575 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121605 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121633 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121751 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121797 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcx4\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.121990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.121839 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.122205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.122007 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.122205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.122092 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.122205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.122177 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.122390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.122247 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.122457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.122381 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.124990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.124957 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.125167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.125063 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.135858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.135830 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcx4\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.142292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.135928 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.245242 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.243998 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:09.563881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.563826 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" podStartSLOduration=1.699532952 podStartE2EDuration="7.56381007s" totalImagesPullingTime="5.864277118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:08.989407614 +0000 UTC m=+789.095196556" watchObservedRunningTime="2026-11-02 01:28:09.56381007 +0000 UTC m=+789.669598984" Nov 02 01:28:09.566821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.566791 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz"] Nov 02 01:28:09.580575 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:28:09.580543 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3409c2a9_1820_4a11_95cd_b05b011a023f.slice/crio-1e18e77aada980184d2f85e92a241d9e862a406d17eba1c5e73676cef788ee70 WatchSource:0}: Error finding container 1e18e77aada980184d2f85e92a241d9e862a406d17eba1c5e73676cef788ee70: Status 404 returned error can't find the container with id 1e18e77aada980184d2f85e92a241d9e862a406d17eba1c5e73676cef788ee70 Nov 02 01:28:09.583691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.583647 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:09.583790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.583744 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:09.583835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.583786 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:09.975453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.975414 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" event={"ID":"3409c2a9-1820-4a11-95cd-b05b011a023f","Type":"ContainerStarted","Data":"d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce"} Nov 02 01:28:09.976017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.975480 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" event={"ID":"3409c2a9-1820-4a11-95cd-b05b011a023f","Type":"ContainerStarted","Data":"1e18e77aada980184d2f85e92a241d9e862a406d17eba1c5e73676cef788ee70"} Nov 02 01:28:09.977835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:09.977814 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q47qx" Nov 02 01:28:10.002848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:10.002791 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podStartSLOduration=2.002775092 podStartE2EDuration="2.002775092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:09.999926843 +0000 UTC m=+790.105715756" watchObservedRunningTime="2026-11-02 01:28:10.002775092 +0000 UTC m=+790.108564067" Nov 02 01:28:10.246987 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:10.246892 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:10.248401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:10.248366 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:10.248565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:10.248434 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:11.254144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:11.253534 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:11.254144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:11.253603 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:12.270276 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:12.270231 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:12.270773 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:12.270304 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:13.258140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:13.249509 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:13.258140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:13.249585 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:14.246461 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:14.246334 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:14.257096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:14.247031 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:15.251590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:15.251549 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:15.252581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:15.252181 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:15.266310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:15.266277 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:28:16.246703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:16.246668 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 01:28:16.247096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:16.247049 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 01:28:18.067728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:18.067691 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:18.105886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:18.105839 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:18.144551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:18.144523 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:28:27.751006 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:28:27.750966 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:28:39.384247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.384212 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq"] Nov 02 01:28:39.390511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.390484 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.406881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.406833 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq"] Nov 02 01:28:39.412056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.412025 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c"] Nov 02 01:28:39.416777 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.416749 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.436677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.436639 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c"] Nov 02 01:28:39.449372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.449338 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.449707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.449684 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.457367 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457338 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.457661 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457637 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.457790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457674 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.457790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457724 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.457790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457752 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.457790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457776 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.457949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457831 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.457949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457855 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.457949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457920 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.458056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.457944 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bkm\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.458056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.458003 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9ks\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.458056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.458034 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.458210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.458065 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.458210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.458103 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.458210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.458182 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.458210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.458207 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.558957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.558916 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.558975 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559002 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559027 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559045 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559070 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559128 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559151 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559200 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559227 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bkm\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559275 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9ks\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559321 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559358 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559395 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559426 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559450 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559496 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559535 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.559572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559572 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.560221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.559820 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.560221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.560022 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.560221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.560138 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.560506 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.560406 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.560506 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.560448 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.560887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.560862 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.561129 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.561088 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.561212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.561151 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.561306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.561280 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.562512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.562467 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.562607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.562538 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.562919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.562899 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.563236 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.563213 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.572093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.572065 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.572633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.572498 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.572680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.572654 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9ks\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.573293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.573264 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bkm\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:39.710957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.710864 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:39.757621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:39.757558 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:40.009154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.009106 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq"] Nov 02 01:28:40.009356 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:28:40.009332 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c247f7d_fd19_4518_9f7c_fb64606ba2dd.slice/crio-70499b79d2598701f0b7d819f2bd615bc42d4cf128415a9feb3c6e22e64b01ea WatchSource:0}: Error finding container 70499b79d2598701f0b7d819f2bd615bc42d4cf128415a9feb3c6e22e64b01ea: Status 404 returned error can't find the container with id 70499b79d2598701f0b7d819f2bd615bc42d4cf128415a9feb3c6e22e64b01ea Nov 02 01:28:40.011900 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.011866 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:40.011989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.011936 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:40.011989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.011964 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:40.086495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.086469 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c"] Nov 02 01:28:40.088303 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:28:40.088277 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a3e4e7_a07d_4e1b_ba4b_ef74f0a3fe11.slice/crio-d46406bf436d069901a48e63b210a73653d13ada1bf2c41d5b97c2975e17b937 WatchSource:0}: Error finding container d46406bf436d069901a48e63b210a73653d13ada1bf2c41d5b97c2975e17b937: Status 404 returned error can't find the container with id d46406bf436d069901a48e63b210a73653d13ada1bf2c41d5b97c2975e17b937 Nov 02 01:28:40.092093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.092008 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:40.092240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.092143 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:40.092240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.092174 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:40.473255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.468756 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" event={"ID":"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11","Type":"ContainerStarted","Data":"c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd"} Nov 02 01:28:40.473255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.468814 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" event={"ID":"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11","Type":"ContainerStarted","Data":"d46406bf436d069901a48e63b210a73653d13ada1bf2c41d5b97c2975e17b937"} Nov 02 01:28:40.473852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.473788 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" event={"ID":"5c247f7d-fd19-4518-9f7c-fb64606ba2dd","Type":"ContainerStarted","Data":"484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e"} Nov 02 01:28:40.473852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.473836 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" event={"ID":"5c247f7d-fd19-4518-9f7c-fb64606ba2dd","Type":"ContainerStarted","Data":"70499b79d2598701f0b7d819f2bd615bc42d4cf128415a9feb3c6e22e64b01ea"} Nov 02 01:28:40.508262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.508202 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podStartSLOduration=1.508182697 podStartE2EDuration="1.508182697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:40.498589409 +0000 UTC m=+820.604378323" watchObservedRunningTime="2026-11-02 01:28:40.508182697 +0000 UTC m=+820.613971612" Nov 02 01:28:40.523056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.522992 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podStartSLOduration=1.5229735039999999 podStartE2EDuration="1.522973504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:40.521581674 +0000 UTC m=+820.627370589" watchObservedRunningTime="2026-11-02 01:28:40.522973504 +0000 UTC m=+820.628762418" Nov 02 01:28:40.711411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.711376 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:40.758155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:40.758047 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:41.712590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:41.712551 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:28:41.713013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:41.712607 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 01:28:41.766142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:41.765707 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:28:41.766142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:41.765780 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 01:28:42.713145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:42.712156 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:28:42.713145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:42.712213 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 01:28:42.784187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:42.762890 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:28:42.784542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:42.784501 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 01:28:43.713296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:43.712517 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:28:43.713296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:43.712580 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 01:28:43.759229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:43.759195 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:28:43.759496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:43.759466 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 01:28:44.711662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:44.711618 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:28:44.711976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:44.711682 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 01:28:44.765588 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:44.765549 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:28:44.766558 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:44.766176 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 01:28:45.711510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:45.711479 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:28:45.711783 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:45.711758 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 01:28:45.764146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:45.762538 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:28:45.764146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:45.762615 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 01:28:46.711536 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:46.711489 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:28:46.712006 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:46.711561 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 01:28:46.758677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:46.758637 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:28:46.758878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:46.758705 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 01:28:47.721838 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:47.721803 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:47.722379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:47.722147 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:47.723308 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:47.723286 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:28:48.050883 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:48.050797 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:48.051348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:48.051330 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:48.052292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:48.052250 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:28:53.936103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:53.936069 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5"] Nov 02 01:28:53.941009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:53.940986 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:53.969181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:53.969153 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5"] Nov 02 01:28:54.095123 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095093 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095173 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095222 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095482 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095540 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095593 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095670 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095734 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095717 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.095800 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.095780 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97gw\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.164111 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.164071 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd"] Nov 02 01:28:54.169794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.169771 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.184731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.184705 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd"] Nov 02 01:28:54.197823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.197759 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.197947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.197831 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.197947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.197878 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.197947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.197922 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97gw\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.197948 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.197995 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.198009 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.198185 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198246 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.198225 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.198529 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.198723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.198710 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.199238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.199211 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.199372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.199338 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.199861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.199818 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.204161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.201478 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.204161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.201953 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.212011 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.211980 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97gw\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.212577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.212554 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.269014 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.268981 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:54.299569 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.299536 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.299687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.299621 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.299687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.299662 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.299763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.299732 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.299940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.299919 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.300050 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.299971 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.300135 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.300056 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26b96\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.300135 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.300076 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.300248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.300138 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.400916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.400875 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401095 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.400923 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401095 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401076 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401150 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401337 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401392 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26b96\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401431 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401487 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401585 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401616 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.401866 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401676 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.402248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401910 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.402248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.401943 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.402623 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.402472 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.405164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.405138 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.406897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.406875 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.413548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.413525 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.416176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.416154 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26b96\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.470296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.470267 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf"] Nov 02 01:28:54.486042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.485874 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf"] Nov 02 01:28:54.486228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.486007 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.494629 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.494605 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:54.561405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.561010 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5"] Nov 02 01:28:54.570814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.570774 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:54.571020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.571000 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:54.571173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.571156 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:54.604009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.603985 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604019 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqgv\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604099 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604170 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604247 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604323 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604338 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604360 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.604446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.604383 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.687141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.687042 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" event={"ID":"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f","Type":"ContainerStarted","Data":"62ae71e1bcdd1e29b3e6ca225c72a6af0ec35fee9821c2cbad5d81731c77369c"} Nov 02 01:28:54.705536 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.705501 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.705536 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.705542 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.705797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.705574 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.705824 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706087 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706620 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706186 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706620 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706262 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706620 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706565 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706785 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706676 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqgv\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706851 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706789 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706851 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706822 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.706948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.706872 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.707230 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.707184 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.707828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.707805 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.708314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.708293 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.709522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.709503 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.735159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.732960 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.735159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.733006 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqgv\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.807977 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.807948 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:54.860670 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.860607 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd"] Nov 02 01:28:54.872719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.871974 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:54.872719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.872066 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:54.872719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:54.872106 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:55.132941 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:28:55.132906 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371fa9aa_2a3a_405e_8f0e_eb05a57b842c.slice/crio-4df249e26cc78fc82a34119ee4d691dd8583c7da62d002517794099f4ca9a143 WatchSource:0}: Error finding container 4df249e26cc78fc82a34119ee4d691dd8583c7da62d002517794099f4ca9a143: Status 404 returned error can't find the container with id 4df249e26cc78fc82a34119ee4d691dd8583c7da62d002517794099f4ca9a143 Nov 02 01:28:55.134104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.134080 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf"] Nov 02 01:28:55.135748 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.135711 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:55.135862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.135775 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:55.135862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.135803 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:28:55.716540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.716492 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" event={"ID":"debef081-f6cd-4537-bc41-e1644e0c5200","Type":"ContainerStarted","Data":"02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e"} Nov 02 01:28:55.716725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.716564 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" event={"ID":"debef081-f6cd-4537-bc41-e1644e0c5200","Type":"ContainerStarted","Data":"be1aae800428e69d5015c6cc0d4790e25fb18ef86cee060c808d9f15fbc50908"} Nov 02 01:28:55.718986 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.718959 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" event={"ID":"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f","Type":"ContainerStarted","Data":"ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054"} Nov 02 01:28:55.721438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.721410 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" event={"ID":"371fa9aa-2a3a-405e-8f0e-eb05a57b842c","Type":"ContainerStarted","Data":"5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b"} Nov 02 01:28:55.721555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.721463 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" event={"ID":"371fa9aa-2a3a-405e-8f0e-eb05a57b842c","Type":"ContainerStarted","Data":"4df249e26cc78fc82a34119ee4d691dd8583c7da62d002517794099f4ca9a143"} Nov 02 01:28:55.760140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.757487 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podStartSLOduration=1.757467506 podStartE2EDuration="1.757467506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.745650329 +0000 UTC m=+835.851439244" watchObservedRunningTime="2026-11-02 01:28:55.757467506 +0000 UTC m=+835.863256425" Nov 02 01:28:55.794525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.794469 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podStartSLOduration=1.794454585 podStartE2EDuration="1.794454585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.793575279 +0000 UTC m=+835.899364193" watchObservedRunningTime="2026-11-02 01:28:55.794454585 +0000 UTC m=+835.900243498" Nov 02 01:28:55.808503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.808468 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:28:55.818922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:55.818879 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podStartSLOduration=2.81886639 podStartE2EDuration="2.81886639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.815938535 +0000 UTC m=+835.921727451" watchObservedRunningTime="2026-11-02 01:28:55.81886639 +0000 UTC m=+835.924655304" Nov 02 01:28:56.302144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.292171 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:28:56.303021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.302990 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.303273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.303239 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:28:56.498143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.497508 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:28:56.502141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.499847 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.502141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.499900 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:28:56.826370 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.826328 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.826590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:56.826396 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:28:57.290144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:57.281342 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.290144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:57.281404 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:28:57.517143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:57.498477 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.517143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:57.498538 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:28:57.831164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:57.814500 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.831164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:57.814558 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:28:57.835737 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:28:57.835697 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:28:58.293144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:58.274808 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:28:58.293144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:58.274867 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:28:58.506144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:58.502237 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:28:58.506144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:58.502320 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:28:58.891407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:58.891367 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:28:58.904289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:58.904237 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:28:59.292210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:59.292175 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.296136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:59.292459 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:28:59.563194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:59.560443 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.563194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:59.560514 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:28:59.831197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:59.830649 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.831197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:28:59.830710 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:29:00.353142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.352888 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.353142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.352958 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:29:00.491822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.491766 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.515550 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.515512 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.515797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.515766 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:29:00.815022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.814982 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.815371 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:00.815324 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:29:01.287145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:01.286915 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.287145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:01.286973 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:29:01.502818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:01.502782 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.504385 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:01.504353 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:29:01.810523 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:01.810481 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.810810 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:01.810782 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:29:02.269686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:02.269651 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.270007 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:02.269966 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 01:29:02.494969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:02.494912 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.495185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:02.494982 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 01:29:02.808800 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:02.808460 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.808800 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:02.808522 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 01:29:03.398498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.379853 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:29:03.398498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.385776 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:29:03.403233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.403202 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:29:03.820696 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.820664 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:29:03.822212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.822191 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:29:03.823584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.823560 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:29:03.824715 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.824686 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:29:03.830167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.828314 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:29:03.830167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:03.828433 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:29:04.050861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:04.050827 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:29:09.475628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.475558 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf"] Nov 02 01:29:09.482043 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.482023 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.494291 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.494263 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf"] Nov 02 01:29:09.637955 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.637922 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638019 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638049 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcr8s\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638149 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638188 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638283 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638359 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638504 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.638577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.638533 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.739897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.739791 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.739924 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.739943 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740019 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740082 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740110 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcr8s\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740175 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740214 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740417 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740275 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740417 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740356 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740397 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740478 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740776 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.740935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.740913 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.742549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.742528 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.743257 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.743236 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.755739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.755715 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcr8s\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.755967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.755913 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:09.826446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:09.826413 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:10.125266 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:10.125226 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10383294_673a_40a9_ab9b_6ac98c648a09.slice/crio-53b6165585927ec47491a65ddbc50b2536f7f61eb3c59b09a3d76fad4185cdf2 WatchSource:0}: Error finding container 53b6165585927ec47491a65ddbc50b2536f7f61eb3c59b09a3d76fad4185cdf2: Status 404 returned error can't find the container with id 53b6165585927ec47491a65ddbc50b2536f7f61eb3c59b09a3d76fad4185cdf2 Nov 02 01:29:10.125424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.125376 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf"] Nov 02 01:29:10.127969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.127936 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:10.128074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.127998 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:10.128074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.128028 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:10.571048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.571014 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" event={"ID":"10383294-673a-40a9-ab9b-6ac98c648a09","Type":"ContainerStarted","Data":"3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0"} Nov 02 01:29:10.571564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.571072 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" event={"ID":"10383294-673a-40a9-ab9b-6ac98c648a09","Type":"ContainerStarted","Data":"53b6165585927ec47491a65ddbc50b2536f7f61eb3c59b09a3d76fad4185cdf2"} Nov 02 01:29:10.595110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.595058 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podStartSLOduration=1.595043674 podStartE2EDuration="1.595043674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:10.593006041 +0000 UTC m=+850.698794957" watchObservedRunningTime="2026-11-02 01:29:10.595043674 +0000 UTC m=+850.700832644" Nov 02 01:29:10.826794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:10.826710 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:11.829248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:11.829214 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:29:11.829669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:11.829267 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 01:29:12.827770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:12.827734 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:29:12.827985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:12.827793 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 01:29:13.827265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:13.827224 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:29:13.827753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:13.827291 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 01:29:14.827218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:14.827181 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:29:14.827439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:14.827276 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 01:29:15.827274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:15.827238 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:29:15.827533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:15.827309 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 01:29:16.827325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:16.827292 4552 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:29:16.827530 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:16.827361 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 01:29:17.832103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:17.832066 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:17.832621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:17.832484 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:17.833271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:17.833251 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:29:24.620445 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.620298 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t"] Nov 02 01:29:24.677048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.676052 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.854335 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.854289 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.854763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.854740 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.855009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.854989 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.855407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.855214 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.861971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.861945 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnb5\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.862317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.862296 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.863825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.863800 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.864071 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.864052 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:24.864338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:24.864320 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.007149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.006172 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t"] Nov 02 01:29:25.079749 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.076219 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq"] Nov 02 01:29:25.095784 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.085382 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211402 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211493 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211557 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211639 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211722 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211770 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211883 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.211939 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212043 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212085 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnb5\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212166 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212197 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212248 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212322 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww95\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212371 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212422 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212557 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.212873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.212606 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.213314 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.213732 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.229212 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq"] Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.232888 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.233527 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.244524 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.245199 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.245782 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.254555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.252816 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.282228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.280797 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnb5\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325390 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325517 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325590 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww95\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325643 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325722 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325768 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325833 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325887 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.325951 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.326500 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.335141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.326782 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.349512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.349477 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.350074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.350050 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.350314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.350288 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.351013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.350991 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.360138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.351398 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.415850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.415808 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.416716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.416690 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww95\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:25.465443 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.465405 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:25.580557 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:25.579452 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:26.528009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:26.444698 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t"] Nov 02 01:29:26.528009 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:26.505519 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac3ae08_3b39_4d9a_952b_c859af24c6a9.slice/crio-716965bed473e69c1afddd075d036144c888f46cc8469c35cfc1ef3ddd55edb7 WatchSource:0}: Error finding container 716965bed473e69c1afddd075d036144c888f46cc8469c35cfc1ef3ddd55edb7: Status 404 returned error can't find the container with id 716965bed473e69c1afddd075d036144c888f46cc8469c35cfc1ef3ddd55edb7 Nov 02 01:29:26.556589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:26.555803 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:26.556589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:26.555902 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:26.556589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:26.555952 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:27.021231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.020760 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" event={"ID":"6ac3ae08-3b39-4d9a-952b-c859af24c6a9","Type":"ContainerStarted","Data":"d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049"} Nov 02 01:29:27.021231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.020831 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" event={"ID":"6ac3ae08-3b39-4d9a-952b-c859af24c6a9","Type":"ContainerStarted","Data":"716965bed473e69c1afddd075d036144c888f46cc8469c35cfc1ef3ddd55edb7"} Nov 02 01:29:27.051156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.051062 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podStartSLOduration=3.051047315 podStartE2EDuration="3.051047315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:27.04699706 +0000 UTC m=+867.152785974" watchObservedRunningTime="2026-11-02 01:29:27.051047315 +0000 UTC m=+867.156836228" Nov 02 01:29:27.155252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.155219 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq"] Nov 02 01:29:27.159728 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:27.159685 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18df2e17_68a5_45d6_826e_abaca94c16b5.slice/crio-e66b2f41cf265d3cee89ed51d041649d72569142aace3441106b15018e2cb120 WatchSource:0}: Error finding container e66b2f41cf265d3cee89ed51d041649d72569142aace3441106b15018e2cb120: Status 404 returned error can't find the container with id e66b2f41cf265d3cee89ed51d041649d72569142aace3441106b15018e2cb120 Nov 02 01:29:27.163751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.163209 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:27.163751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.163302 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:27.163751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.163344 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:27.467528 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.467485 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:27.468568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.468537 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:27.468704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:27.468606 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:27.762441 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:29:27.762398 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:29:28.036229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.033092 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" event={"ID":"18df2e17-68a5-45d6-826e-abaca94c16b5","Type":"ContainerStarted","Data":"0e029a7fd78c7661a5251423a0541d1436bc4c06bf9b2de8455a9d59515c86b8"} Nov 02 01:29:28.036229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.033235 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" event={"ID":"18df2e17-68a5-45d6-826e-abaca94c16b5","Type":"ContainerStarted","Data":"e66b2f41cf265d3cee89ed51d041649d72569142aace3441106b15018e2cb120"} Nov 02 01:29:28.133318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.133262 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podStartSLOduration=4.133240863 podStartE2EDuration="4.133240863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:28.078659359 +0000 UTC m=+868.184448274" watchObservedRunningTime="2026-11-02 01:29:28.133240863 +0000 UTC m=+868.239029768" Nov 02 01:29:28.507775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.507722 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:28.507997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.507809 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:28.581751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.581705 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:28.593692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.582087 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 01:29:28.594052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:28.594022 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 01:29:29.469559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:29.469521 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:29.470247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:29.470209 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:29.606592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:29.606552 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 01:29:29.606791 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:29.606614 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 01:29:30.472142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:30.471733 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:30.472142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:30.471800 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:30.600143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:30.599489 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 01:29:30.600143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:30.599550 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 01:29:31.477707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:31.476568 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:31.477707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:31.476631 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:31.588140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:31.583135 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 01:29:31.588140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:31.583217 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 01:29:32.485147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:32.466359 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:32.485147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:32.466442 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:32.582551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:32.582511 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 01:29:32.585788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:32.585753 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 01:29:33.473634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:33.473596 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 01:29:33.473940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:33.473894 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 01:29:33.580650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:33.580608 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 01:29:33.581067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:33.580691 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 01:29:35.005548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:35.005514 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:35.006223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:35.006190 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:35.007046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:35.007020 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:29:35.109495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:35.109465 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:35.110971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:35.110943 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:35.115322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:35.115248 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:29:39.087453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.087418 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p"] Nov 02 01:29:39.094135 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.094087 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.129854 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.129823 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p"] Nov 02 01:29:39.214960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.214925 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.214978 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215100 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215180 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8pt\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215306 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215350 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215412 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215492 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.215540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.215529 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317236 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317202 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317281 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317320 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8pt\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317378 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317412 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317432 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317496 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317532 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317648 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317795 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.317878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317802 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.318029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.317998 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.318181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.318048 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.318811 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.318789 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.320038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.320014 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.320163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.320082 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.331033 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.331010 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8pt\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.331160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.331069 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.432860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.432772 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:39.531229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.531095 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7"] Nov 02 01:29:39.543249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.541669 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.547642 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.547611 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr"] Nov 02 01:29:39.558447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.557735 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7"] Nov 02 01:29:39.558447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.557900 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.567237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.566580 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr"] Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.635532 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.635615 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t4nh\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.635690 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.635842 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.635892 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.635964 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636039 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636161 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636251 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636302 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636405 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636436 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636460 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636499 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636525 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636548 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncln\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636574 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.636712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.636601 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.738522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738486 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738541 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738603 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738629 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738645 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738668 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738692 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.738724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738718 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncln\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738744 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738759 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738827 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738891 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t4nh\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738964 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.738998 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739030 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739068 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739093 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739134 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739220 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739255 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739271 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739335 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739457 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739583 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.739742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739676 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739824 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.739886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.739824 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.740077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.740054 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.741833 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.741805 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.742342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.742316 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.742460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.742394 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.742602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.742585 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.754348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.753780 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t4nh\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.754348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.754275 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.754348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.754299 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncln\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.754714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.754694 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:39.757325 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:39.757301 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4598ffff_e996_43b2_b1cd_c87e754d3cfd.slice/crio-948e27cb6844baa0dbb2a2c6a1c7b65f221991ea2f396e5c91bfe6a2423231e9 WatchSource:0}: Error finding container 948e27cb6844baa0dbb2a2c6a1c7b65f221991ea2f396e5c91bfe6a2423231e9: Status 404 returned error can't find the container with id 948e27cb6844baa0dbb2a2c6a1c7b65f221991ea2f396e5c91bfe6a2423231e9 Nov 02 01:29:39.757325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.757317 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p"] Nov 02 01:29:39.760673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.760641 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:39.760795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.760715 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:39.760795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.760762 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:39.868519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.868486 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:39.913777 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:39.913737 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:40.185603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.185576 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7"] Nov 02 01:29:40.186276 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:40.186251 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec77da6_3db7_4bde_951a_d829e9706a9c.slice/crio-e391274b9434a870eefb70602b12dd7840f1df6d5f48cdc149c7f728425e5f89 WatchSource:0}: Error finding container e391274b9434a870eefb70602b12dd7840f1df6d5f48cdc149c7f728425e5f89: Status 404 returned error can't find the container with id e391274b9434a870eefb70602b12dd7840f1df6d5f48cdc149c7f728425e5f89 Nov 02 01:29:40.189651 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.189603 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:40.189761 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.189700 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:40.189834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.189765 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:40.258471 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:40.258440 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee47da7_374c_4419_a443_013a36e93790.slice/crio-a24d06c40b3ef01a488fea557ae69c6ef34620a3aaf27aaf5eadd5606ca53764 WatchSource:0}: Error finding container a24d06c40b3ef01a488fea557ae69c6ef34620a3aaf27aaf5eadd5606ca53764: Status 404 returned error can't find the container with id a24d06c40b3ef01a488fea557ae69c6ef34620a3aaf27aaf5eadd5606ca53764 Nov 02 01:29:40.262052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.262018 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:40.262213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.262089 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:40.262213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.262138 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:40.264848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.264820 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr"] Nov 02 01:29:40.296054 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.296017 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" event={"ID":"4598ffff-e996-43b2-b1cd-c87e754d3cfd","Type":"ContainerStarted","Data":"853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42"} Nov 02 01:29:40.296238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.296083 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" event={"ID":"4598ffff-e996-43b2-b1cd-c87e754d3cfd","Type":"ContainerStarted","Data":"948e27cb6844baa0dbb2a2c6a1c7b65f221991ea2f396e5c91bfe6a2423231e9"} Nov 02 01:29:40.299305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.299270 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" event={"ID":"7ec77da6-3db7-4bde-951a-d829e9706a9c","Type":"ContainerStarted","Data":"4293a9594c01b74313af985fcdf59ce3820942aab20d50ab18d082ab2d70c294"} Nov 02 01:29:40.299486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.299328 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" event={"ID":"7ec77da6-3db7-4bde-951a-d829e9706a9c","Type":"ContainerStarted","Data":"e391274b9434a870eefb70602b12dd7840f1df6d5f48cdc149c7f728425e5f89"} Nov 02 01:29:40.303245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.303219 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" event={"ID":"7ee47da7-374c-4419-a443-013a36e93790","Type":"ContainerStarted","Data":"a24d06c40b3ef01a488fea557ae69c6ef34620a3aaf27aaf5eadd5606ca53764"} Nov 02 01:29:40.357003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.355232 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podStartSLOduration=1.35520928 podStartE2EDuration="1.35520928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.354078549 +0000 UTC m=+880.459867464" watchObservedRunningTime="2026-11-02 01:29:40.35520928 +0000 UTC m=+880.460998195" Nov 02 01:29:40.373424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.369930 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podStartSLOduration=1.3699150119999999 podStartE2EDuration="1.369915012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.331641395 +0000 UTC m=+880.437430310" watchObservedRunningTime="2026-11-02 01:29:40.369915012 +0000 UTC m=+880.475703926" Nov 02 01:29:40.433466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.433412 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:40.435441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.435153 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:40.435441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.435204 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:40.885298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:40.885253 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:41.350304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.350253 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" event={"ID":"7ee47da7-374c-4419-a443-013a36e93790","Type":"ContainerStarted","Data":"0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797"} Nov 02 01:29:41.425142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.422526 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podStartSLOduration=2.422499782 podStartE2EDuration="2.422499782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:41.389747505 +0000 UTC m=+881.495536419" watchObservedRunningTime="2026-11-02 01:29:41.422499782 +0000 UTC m=+881.528288700" Nov 02 01:29:41.434564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.434533 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:41.434760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.434580 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:41.875795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.875759 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:41.876190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.876159 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:41.923141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.916585 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:41.923141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.918633 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:41.923141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:41.918693 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:42.433198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:42.433148 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:42.433589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:42.433222 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:42.870055 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:42.869567 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:42.870055 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:42.869644 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:42.918725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:42.918678 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:42.919025 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:42.918970 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:43.524169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:43.457289 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:43.524169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:43.457393 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:44.022623 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.022583 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.063559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.063519 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:44.081602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.024661 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.085151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.081860 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:44.438198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.434709 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.438198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.434789 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:44.900567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.900523 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.901076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.900603 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:44.970153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.967597 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.970153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:44.967675 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:45.433806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:45.433777 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:45.434056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:45.434037 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:45.921143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:45.918555 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:45.921143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:45.918619 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:45.921143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:45.919066 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:45.921143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:45.919106 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:46.434175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:46.434139 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:46.434441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:46.434415 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:46.879166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:46.879082 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:46.879684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:46.879650 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:46.916625 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:46.916588 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:46.916957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:46.916905 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:47.433544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:47.433495 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:29:47.434091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:47.433582 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 01:29:47.876559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:47.876521 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:29:47.878452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:47.878369 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 01:29:47.920602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:47.920536 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:29:47.920911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:47.920879 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 01:29:48.445438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.445407 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:48.447658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.447634 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:48.448780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.448764 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:29:48.874504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.874475 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:48.874979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.874957 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:48.875714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.875692 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:29:48.919842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.919802 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:48.920098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.920075 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:48.921016 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:48.920984 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:29:54.028637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.028607 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6"] Nov 02 01:29:54.035658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.035634 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.092155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.082438 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.092155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.082524 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.092155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.082562 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.103141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.093238 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.103141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.093302 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.103141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.093368 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.103141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.093452 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkjb\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.103141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.093653 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.103141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.093705 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.140992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.140959 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6"] Nov 02 01:29:54.199633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.199592 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkjb\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.199836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.199807 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.199886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.199867 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.199993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.199972 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.200250 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200231 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.200421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200402 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.200605 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200589 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.200719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200706 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.200862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200843 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.200932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200850 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.201460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.201440 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.201624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.201609 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.201707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200324 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.202069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.200705 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.203699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.203679 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.204266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.204249 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.212362 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.212341 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkjb\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.215272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.215247 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.267012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.266976 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw"] Nov 02 01:29:54.274130 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.274086 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.287411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.287337 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw"] Nov 02 01:29:54.403227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403193 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403231 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchkv\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403401 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403430 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403527 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403648 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403634 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403660 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403711 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.403766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.403753 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.419147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.419088 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.513263 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.513787 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.513858 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.513896 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.513955 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.513994 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchkv\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.514203 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.514239 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.514325 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.514496 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.514785 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.515159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.515093 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.516413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.515228 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.516413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.515269 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.518657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.518629 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.519403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.519384 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.526815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.526704 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchkv\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.526815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.526778 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.599536 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.599448 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:54.721812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.721785 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6"] Nov 02 01:29:54.722583 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:54.722561 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode281771e_1091_4348_b47f_a7c5152c9cdc.slice/crio-f4679f3f29822c03b36b4a116a6695f25ed89cfe7ecacdeb6f92e532a4339d2a WatchSource:0}: Error finding container f4679f3f29822c03b36b4a116a6695f25ed89cfe7ecacdeb6f92e532a4339d2a: Status 404 returned error can't find the container with id f4679f3f29822c03b36b4a116a6695f25ed89cfe7ecacdeb6f92e532a4339d2a Nov 02 01:29:54.725894 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.725850 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:54.725971 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.725948 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:54.726010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.725993 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:54.891888 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:29:54.891847 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1139d23_22e0_4437_be47_a1f225d775b9.slice/crio-a39e8644db45acbb7335a4e874814b8d0e9a9b731e3161aa6f444280d1ddc466 WatchSource:0}: Error finding container a39e8644db45acbb7335a4e874814b8d0e9a9b731e3161aa6f444280d1ddc466: Status 404 returned error can't find the container with id a39e8644db45acbb7335a4e874814b8d0e9a9b731e3161aa6f444280d1ddc466 Nov 02 01:29:54.892367 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.892342 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw"] Nov 02 01:29:54.895333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.895299 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:54.895429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.895373 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:54.895429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:54.895401 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:29:55.263160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.257190 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" event={"ID":"e281771e-1091-4348-b47f-a7c5152c9cdc","Type":"ContainerStarted","Data":"81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a"} Nov 02 01:29:55.263160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.257286 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" event={"ID":"e281771e-1091-4348-b47f-a7c5152c9cdc","Type":"ContainerStarted","Data":"f4679f3f29822c03b36b4a116a6695f25ed89cfe7ecacdeb6f92e532a4339d2a"} Nov 02 01:29:55.289080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.289048 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" event={"ID":"f1139d23-22e0-4437-be47-a1f225d775b9","Type":"ContainerStarted","Data":"62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845"} Nov 02 01:29:55.289291 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.289219 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" event={"ID":"f1139d23-22e0-4437-be47-a1f225d775b9","Type":"ContainerStarted","Data":"a39e8644db45acbb7335a4e874814b8d0e9a9b731e3161aa6f444280d1ddc466"} Nov 02 01:29:55.327455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.327378 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podStartSLOduration=1.327359204 podStartE2EDuration="1.327359204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.287094209 +0000 UTC m=+895.392883137" watchObservedRunningTime="2026-11-02 01:29:55.327359204 +0000 UTC m=+895.433148118" Nov 02 01:29:55.327667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.327538 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podStartSLOduration=1.3275282609999999 podStartE2EDuration="1.327528261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.323213475 +0000 UTC m=+895.429002403" watchObservedRunningTime="2026-11-02 01:29:55.327528261 +0000 UTC m=+895.433317175" Nov 02 01:29:55.419529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.419494 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:29:55.420733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.420708 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:29:55.420832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.420762 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:29:55.466927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.466887 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:29:55.617204 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.613688 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:29:55.617552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.617522 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:29:55.617741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:55.617706 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:29:56.421142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:56.420841 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:29:56.421142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:56.420919 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:29:56.603144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:56.602521 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:29:56.603144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:56.602587 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:29:57.427573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:57.427534 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:29:57.429331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:57.429297 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:29:57.612148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:57.575985 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.626142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:57.619330 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.626142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:57.622523 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:29:57.626142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:57.622574 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:29:58.188383 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:29:58.188337 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:29:58.233319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.232980 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Nov 02 01:29:58.430283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.419709 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:29:58.458652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.458546 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:29:58.600144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.599889 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.600358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.600291 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:29:58.709205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.701001 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 02 01:29:58.709205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.701940 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:29:58.716873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.716833 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.780214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.780175 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:29:58.794216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.794175 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:29:58.960955 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:58.960792 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.461982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.460622 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:29:59.461982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.460684 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:29:59.616145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.607732 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.666421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.666382 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:29:59.666710 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.666680 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:29:59.753209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.709231 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.781502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.781456 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.798506 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:29:59.798468 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:30:00.483193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:00.472393 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:00.483193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:00.472509 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:00.677146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:00.628508 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 01:30:01.477145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.464394 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Nov 02 01:30:01.477145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:00.916949 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.537151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.512375 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:01.538343 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.538308 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:01.630793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.537326 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:01.631162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.631104 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:01.631462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:00.782734 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.631667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.538643 4552 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.232439193s: [/var/lib/containers/storage/overlay/93aba21acd5cb4f456d852b03dfa995a47a9890639705619ddf72974c40ab79f/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcfgw_4809d979-ef70-435c-b1e5-e403fa49988d/sbdb/0.log]; will not log again for this container unless duration exceeds 2s Nov 02 01:30:01.876424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.874781 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kb25x" podUID="a31aa41f-6bdd-4401-b45b-8a7e863305ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.921990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.921935 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.978066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.978018 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:01.978466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:01.978408 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:02.488146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.469940 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 01:30:02.567136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.567074 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:02.682248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.682199 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:02.682466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.633205 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 02 01:30:02.689156 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:30:02.586718 4552 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.137s" Nov 02 01:30:02.689156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.609985 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:02.689156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.686049 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:02.963396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:02.963267 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.442141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.441568 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:03.442141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.441631 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:03.483665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.483622 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.631162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.610714 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:03.631162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.610775 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:03.657319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.657269 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.657621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.657574 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.698892 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.698780 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.791381 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.791230 4552 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.791381 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.791302 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.813337 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.813305 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.813612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.813584 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.982144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:03.972248 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 01:30:04.116322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.116263 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:04.116625 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.116579 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.193846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.193235 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:04.193846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.193303 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.515145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.505209 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.515145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.506647 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:04.515145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.506697 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:04.568189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.555175 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:04.568189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.555272 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.600767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.600731 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:04.601235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.601201 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:04.725168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.721239 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.973486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:04.973440 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.113149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.113099 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:30:05.121401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.121369 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:30:05.124512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.124486 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:30:05.421217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.421098 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:05.421217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.421186 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:05.607141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.602510 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:05.607141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:05.602573 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:06.420399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:06.420362 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:06.420867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:06.420840 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:06.612098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:06.612064 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:06.612436 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:06.612404 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:07.420295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:07.420257 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:07.420486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:07.420313 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:07.600696 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:07.600657 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:07.600906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:07.600709 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:08.430714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:08.430675 4552 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.440308 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:08.440231 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 01:30:08.608174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:08.608138 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.608511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:08.608470 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 01:30:09.419655 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:09.419595 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:30:09.600286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:09.600247 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:30:10.222413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:10.222380 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:30:10.225633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:10.225607 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:30:10.601496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:10.601238 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:10.601496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:10.601312 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:11.771753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:11.771499 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:11.772871 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:11.772812 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:12.077010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:12.076914 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:30:12.078418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:12.078395 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:30:22.144705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.144668 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7"] Nov 02 01:30:22.162679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.162561 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.165963 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.165937 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7"] Nov 02 01:30:22.169633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.169606 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-854ec26e-serving-cert\"" Nov 02 01:30:22.219390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.219357 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shsfv\" (UniqueName: \"kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.219589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.219443 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.219589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.219471 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.219589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.219500 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.321224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.321190 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shsfv\" (UniqueName: \"kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.321428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.321265 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.321428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.321285 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.321428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.321309 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.321977 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.321952 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.323973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.323954 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.324103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.324083 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.334315 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.334286 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shsfv\" (UniqueName: \"kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv\") pod \"maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:22.513230 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:22.512619 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:23.050145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.044808 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7"] Nov 02 01:30:23.119348 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:30:23.108269 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566153ff_2431_4a14_b302_67565cdd2be3.slice/crio-60b499549c2430e1a2dc5467eb01b93d628943e63c3610c7a0b670a0d124467e WatchSource:0}: Error finding container 60b499549c2430e1a2dc5467eb01b93d628943e63c3610c7a0b670a0d124467e: Status 404 returned error can't find the container with id 60b499549c2430e1a2dc5467eb01b93d628943e63c3610c7a0b670a0d124467e Nov 02 01:30:23.119348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.119322 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:30:23.232960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.232917 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" event={"ID":"566153ff-2431-4a14-b302-67565cdd2be3","Type":"ContainerStarted","Data":"60b499549c2430e1a2dc5467eb01b93d628943e63c3610c7a0b670a0d124467e"} Nov 02 01:30:23.436107 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.436024 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz"] Nov 02 01:30:23.466299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.466268 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.468530 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.468424 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz"] Nov 02 01:30:23.473452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.473429 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2f21c877-serving-cert\"" Nov 02 01:30:23.476666 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.476641 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.477004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.476970 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwt5\" (UniqueName: \"kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.477166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.477082 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.477264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.477168 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.581260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.581227 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwt5\" (UniqueName: \"kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.582091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.582048 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.584034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.583997 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.584528 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.584493 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.585410 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:30:23.584395 4552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-serving-cert: secret "maas-api-e2e-ait-2f21c877-serving-cert" not found Nov 02 01:30:23.595949 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:30:23.585683 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls podName:ca5bd001-5b54-453d-908a-0bde329e4100 nodeName:}" failed. No retries permitted until 2026-11-02 01:30:24.085657681 +0000 UTC m=+924.191446595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls") pod "maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" (UID: "ca5bd001-5b54-453d-908a-0bde329e4100") : secret "maas-api-e2e-ait-2f21c877-serving-cert" not found Nov 02 01:30:23.598188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.583852 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.633069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.633024 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwt5\" (UniqueName: \"kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:23.662145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:23.662005 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:24.131873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.131784 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:24.135354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.135318 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\") pod \"maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:24.244615 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.244577 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" event={"ID":"566153ff-2431-4a14-b302-67565cdd2be3","Type":"ContainerStarted","Data":"55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579"} Nov 02 01:30:24.245090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.244848 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:24.255446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.255414 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb"] Nov 02 01:30:24.261860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.261833 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.315335 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.300517 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb"] Nov 02 01:30:24.378196 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.345280 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.398318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.398229 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.398873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.398847 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.411317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.411286 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.433271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.433231 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.445382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.445339 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nbpt\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.445729 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.445691 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.445938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.445920 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.446152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.446132 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.450138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.396456 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:24.470507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.355527 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" podStartSLOduration=1.7452058049999999 podStartE2EDuration="2.355505575s" totalImagesPullingTime="610.29977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:24.283574228 +0000 UTC m=+924.389363131" watchObservedRunningTime="2026-11-02 01:30:24.355505575 +0000 UTC m=+924.461294490" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.568575 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.568629 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.568767 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.568884 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.568951 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.569067 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nbpt\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.569153 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.569215 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.569294 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.574999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.569838 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.598384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.598340 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.605141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.600724 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.607191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.607161 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.610691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.610630 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.624406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.621280 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.624406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.621852 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.661683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.661652 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.683677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.675747 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nbpt\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.819208 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.807069 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:24.976604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:24.973956 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz"] Nov 02 01:30:24.977894 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:30:24.977857 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5bd001_5b54_453d_908a_0bde329e4100.slice/crio-57c310665bd10e41d548bceed332f6913df7a83e6a22f6b8f72d0ee215c9927a WatchSource:0}: Error finding container 57c310665bd10e41d548bceed332f6913df7a83e6a22f6b8f72d0ee215c9927a: Status 404 returned error can't find the container with id 57c310665bd10e41d548bceed332f6913df7a83e6a22f6b8f72d0ee215c9927a Nov 02 01:30:25.350406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:25.339754 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb"] Nov 02 01:30:25.433338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:25.431443 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:30:25.433338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:25.431542 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:30:25.433338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:25.431580 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:30:25.446282 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:25.446254 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" event={"ID":"ca5bd001-5b54-453d-908a-0bde329e4100","Type":"ContainerStarted","Data":"57c310665bd10e41d548bceed332f6913df7a83e6a22f6b8f72d0ee215c9927a"} Nov 02 01:30:26.458708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.458656 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" event={"ID":"ca5bd001-5b54-453d-908a-0bde329e4100","Type":"ContainerStarted","Data":"be9ceca3783e48aacf4d86dc1e590d369cd4774e5336df45b4538b58c8cac639"} Nov 02 01:30:26.459272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.458895 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:26.460591 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.460569 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" event={"ID":"a8ee346c-6716-419e-984f-016735e0c703","Type":"ContainerStarted","Data":"eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1"} Nov 02 01:30:26.460726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.460604 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" event={"ID":"a8ee346c-6716-419e-984f-016735e0c703","Type":"ContainerStarted","Data":"bc079f1cf30afa8df8180017a19751301cc4e830f74e547dc57011e5963c9d7f"} Nov 02 01:30:26.508322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.508212 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" podStartSLOduration=2.949861929 podStartE2EDuration="3.508184516s" totalImagesPullingTime="558.322587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:26.501625929 +0000 UTC m=+926.607414844" watchObservedRunningTime="2026-11-02 01:30:26.508184516 +0000 UTC m=+926.613973489" Nov 02 01:30:26.560167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.554971 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podStartSLOduration=2.554949306 podStartE2EDuration="2.554949306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:26.554456717 +0000 UTC m=+926.660245647" watchObservedRunningTime="2026-11-02 01:30:26.554949306 +0000 UTC m=+926.660738219" Nov 02 01:30:26.820454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.820426 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:26.821820 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.821789 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:26.821935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:26.821839 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:27.794050 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:30:27.793453 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:30:27.820823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:27.820783 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:27.820989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:27.820841 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:28.820467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:28.820429 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:28.820880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:28.820495 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:29.820267 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:29.820227 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:29.820456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:29.820300 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:30.472628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:30.472596 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:30:30.828348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:30.828257 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:30.828684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:30.828656 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:31.820287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:31.820253 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:31.820691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:31.820318 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:32.499574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.499544 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:30:32.820735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.820636 4552 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:30:32.821369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.820740 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 01:30:32.931884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.931847 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6"] Nov 02 01:30:32.938079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.938051 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:32.956185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.950995 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6c45ad5e-serving-cert\"" Nov 02 01:30:32.956185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:32.952192 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6"] Nov 02 01:30:33.086287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.080172 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.086649 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.086602 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq82b\" (UniqueName: \"kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.087897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.087866 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.088180 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.088108 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.190262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.190229 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.190487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.190283 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq82b\" (UniqueName: \"kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.190573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.190528 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.190751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.190729 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.191568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.191545 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.193772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.193739 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.197760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.197730 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.207042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.207010 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq82b\" (UniqueName: \"kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b\") pod \"maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.285060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.284084 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:33.754579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:33.754545 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6"] Nov 02 01:30:34.754145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:34.748087 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:34.754145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:34.749760 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:34.800224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:34.796429 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:30:34.859161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:34.844694 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" event={"ID":"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e","Type":"ContainerStarted","Data":"7159438309a7ea2dd02940c2b5d21032c7d0ac7904b1f86552c5d0fa7524b793"} Nov 02 01:30:35.667143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:35.666273 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.222284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:36.222244 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.811216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:36.735313 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.811216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:36.744676 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.811216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:36.765415 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:30:36.892679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:36.892639 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" event={"ID":"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e","Type":"ContainerStarted","Data":"2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c"} Nov 02 01:30:36.929229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:36.929196 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:37.231298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:37.231214 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.363325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:37.339652 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:30:37.743896 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:37.743853 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.745092 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:37.745068 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:30:37.754148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:37.747671 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.982153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:37.975243 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" podStartSLOduration=5.310331411 podStartE2EDuration="5.975219923s" totalImagesPullingTime="664.888512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:37.256413804 +0000 UTC m=+937.362202721" watchObservedRunningTime="2026-11-02 01:30:37.975219923 +0000 UTC m=+938.081008838" Nov 02 01:30:38.503488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:38.503447 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Nov 02 01:30:38.596717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:38.515246 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 01:30:38.754353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:38.754248 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.846715 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:38.846676 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.998197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:38.997744 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Nov 02 01:30:39.599297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:39.599247 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.788097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:39.788055 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.848342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:39.848281 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 01:30:40.667504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:40.667433 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.806224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:40.803412 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.882151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:40.868612 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 01:30:41.152907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:41.151223 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.698189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:41.697247 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.811182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:41.811105 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.051140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:42.050328 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 01:30:42.703141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:42.702641 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:30:42.703671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:42.701931 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 01:30:43.212146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:43.184757 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 01:30:43.385279 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:43.385228 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:43.402590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:43.402527 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.573520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:43.572617 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:30:43.713265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:43.713216 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.827649 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:43.827541 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.239149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:44.217867 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.738156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:44.733295 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.918146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:44.913568 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:30:45.145304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:45.131177 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:30:48.837467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:48.837434 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb"] Nov 02 01:30:48.857533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:48.857505 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:48.886604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:48.886572 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb"] Nov 02 01:30:48.893386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:48.893353 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-616e3ed7-serving-cert\"" Nov 02 01:30:49.011679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.011636 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.012061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.012032 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.012433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.012402 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.012798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.012773 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frhh\" (UniqueName: \"kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.114462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.114373 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frhh\" (UniqueName: \"kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.114651 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.114475 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.114651 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.114543 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.114771 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.114722 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.115554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.115522 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.117490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.117464 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.117902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.117877 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.126655 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.126618 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frhh\" (UniqueName: \"kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh\") pod \"maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.198368 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.198335 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:49.562622 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.562554 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb"] Nov 02 01:30:49.963640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:49.963547 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" event={"ID":"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd","Type":"ContainerStarted","Data":"666f16ab95e9592694635dccc7f3467a181da08a1d9a5fde86e27f12e3ac7e0c"} Nov 02 01:30:50.976607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:50.976571 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" event={"ID":"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd","Type":"ContainerStarted","Data":"7222dfc5ec04d33a42ac680044c9d1435234f1ff7e5c8c694d987583cea715ec"} Nov 02 01:30:50.977039 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:50.976817 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:50.998048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:50.997997 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" podStartSLOduration=2.504371722 podStartE2EDuration="2.997983604s" totalImagesPullingTime="493.611882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:50.995493689 +0000 UTC m=+951.101282603" watchObservedRunningTime="2026-11-02 01:30:50.997983604 +0000 UTC m=+951.103772517" Nov 02 01:30:56.992126 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:30:56.992079 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:30:57.754264 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:30:57.754220 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:31:04.131205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.131168 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7"] Nov 02 01:31:04.131690 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.131410 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" podUID="566153ff-2431-4a14-b302-67565cdd2be3" containerName="maas-api" containerID="cri-o://55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579" gracePeriod=30 Nov 02 01:31:04.293372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.293335 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz"] Nov 02 01:31:04.293835 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.293798 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" podUID="ca5bd001-5b54-453d-908a-0bde329e4100" containerName="maas-api" containerID="cri-o://be9ceca3783e48aacf4d86dc1e590d369cd4774e5336df45b4538b58c8cac639" gracePeriod=30 Nov 02 01:31:04.771756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.764457 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:31:04.788238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.785691 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv\" (UniqueName: \"kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv\") pod \"566153ff-2431-4a14-b302-67565cdd2be3\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " Nov 02 01:31:04.788238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.785755 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca\") pod \"566153ff-2431-4a14-b302-67565cdd2be3\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " Nov 02 01:31:04.788238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.785809 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls\") pod \"566153ff-2431-4a14-b302-67565cdd2be3\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " Nov 02 01:31:04.788238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.786037 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs\") pod \"566153ff-2431-4a14-b302-67565cdd2be3\" (UID: \"566153ff-2431-4a14-b302-67565cdd2be3\") " Nov 02 01:31:04.789840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.789806 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca" pod "566153ff-2431-4a14-b302-67565cdd2be3" (UID: "566153ff-2431-4a14-b302-67565cdd2be3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:04.822401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.819696 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs" pod "566153ff-2431-4a14-b302-67565cdd2be3" (UID: "566153ff-2431-4a14-b302-67565cdd2be3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:04.832408 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.832350 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv" pod "566153ff-2431-4a14-b302-67565cdd2be3" (UID: "566153ff-2431-4a14-b302-67565cdd2be3"). InnerVolumeSpecName "kube-api-access-shsfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:04.833094 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.833051 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls" pod "566153ff-2431-4a14-b302-67565cdd2be3" (UID: "566153ff-2431-4a14-b302-67565cdd2be3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:04.968144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.967188 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:04.968144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.967224 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shsfv\" (UniqueName: \"kubernetes.io/projected/566153ff-2431-4a14-b302-67565cdd2be3-kube-api-access-shsfv\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:04.968144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.967241 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/566153ff-2431-4a14-b302-67565cdd2be3-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:04.968144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:04.967254 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/566153ff-2431-4a14-b302-67565cdd2be3-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.307684 4552 generic.go:356] "Generic (PLEG): container finished" podID="ca5bd001-5b54-453d-908a-0bde329e4100" containerID="be9ceca3783e48aacf4d86dc1e590d369cd4774e5336df45b4538b58c8cac639" exitCode=0 Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.308033 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" event={"ID":"ca5bd001-5b54-453d-908a-0bde329e4100","Type":"ContainerDied","Data":"be9ceca3783e48aacf4d86dc1e590d369cd4774e5336df45b4538b58c8cac639"} Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.310759 4552 generic.go:356] "Generic (PLEG): container finished" podID="566153ff-2431-4a14-b302-67565cdd2be3" containerID="55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579" exitCode=0 Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.310871 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" event={"ID":"566153ff-2431-4a14-b302-67565cdd2be3","Type":"ContainerDied","Data":"55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579"} Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.310901 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.310941 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7" event={"ID":"566153ff-2431-4a14-b302-67565cdd2be3","Type":"ContainerDied","Data":"60b499549c2430e1a2dc5467eb01b93d628943e63c3610c7a0b670a0d124467e"} Nov 02 01:31:05.312213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.310982 4552 scope.go:122] "RemoveContainer" containerID="55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579" Nov 02 01:31:05.340897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.340843 4552 scope.go:122] "RemoveContainer" containerID="55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579" Nov 02 01:31:05.341217 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:31:05.341189 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579\": container with ID starting with 55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579 not found: ID does not exist" containerID="55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579" Nov 02 01:31:05.341323 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.341224 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579"} err="failed to get container status \"55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579\": rpc error: code = NotFound desc = could not find container \"55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579\": container with ID starting with 55cf397b4045bb0b81b4bdd1cd30248ca09c3b30e825a28cbfb06feaa66db579 not found: ID does not exist" Nov 02 01:31:05.343341 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.343290 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:31:05.360260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.360235 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7"] Nov 02 01:31:05.368317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.368291 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ec26e-849f8cd5c-n7bp7"] Nov 02 01:31:05.405940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.405906 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs\") pod \"ca5bd001-5b54-453d-908a-0bde329e4100\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " Nov 02 01:31:05.406136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.405949 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca\") pod \"ca5bd001-5b54-453d-908a-0bde329e4100\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " Nov 02 01:31:05.406136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.406087 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\") pod \"ca5bd001-5b54-453d-908a-0bde329e4100\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " Nov 02 01:31:05.406237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.406140 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5\" (UniqueName: \"kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5\") pod \"ca5bd001-5b54-453d-908a-0bde329e4100\" (UID: \"ca5bd001-5b54-453d-908a-0bde329e4100\") " Nov 02 01:31:05.406399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.406367 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca" pod "ca5bd001-5b54-453d-908a-0bde329e4100" (UID: "ca5bd001-5b54-453d-908a-0bde329e4100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:05.407014 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.406989 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bd001-5b54-453d-908a-0bde329e4100-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:05.408521 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.408491 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls" pod "ca5bd001-5b54-453d-908a-0bde329e4100" (UID: "ca5bd001-5b54-453d-908a-0bde329e4100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:05.408925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.408899 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs" pod "ca5bd001-5b54-453d-908a-0bde329e4100" (UID: "ca5bd001-5b54-453d-908a-0bde329e4100"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:05.409028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.408944 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5" pod "ca5bd001-5b54-453d-908a-0bde329e4100" (UID: "ca5bd001-5b54-453d-908a-0bde329e4100"). InnerVolumeSpecName "kube-api-access-wfwt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:05.514273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.513365 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:05.514273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.513402 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5bd001-5b54-453d-908a-0bde329e4100-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:05.514273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:05.513415 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfwt5\" (UniqueName: \"kubernetes.io/projected/ca5bd001-5b54-453d-908a-0bde329e4100-kube-api-access-wfwt5\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:06.052198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:06.052158 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566153ff-2431-4a14-b302-67565cdd2be3" path="/var/lib/kubelet/pods/566153ff-2431-4a14-b302-67565cdd2be3/volumes" Nov 02 01:31:06.368203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:06.367402 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" event={"ID":"ca5bd001-5b54-453d-908a-0bde329e4100","Type":"ContainerDied","Data":"57c310665bd10e41d548bceed332f6913df7a83e6a22f6b8f72d0ee215c9927a"} Nov 02 01:31:06.368203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:06.367475 4552 scope.go:122] "RemoveContainer" containerID="be9ceca3783e48aacf4d86dc1e590d369cd4774e5336df45b4538b58c8cac639" Nov 02 01:31:06.368203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:06.367595 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz" Nov 02 01:31:06.427754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:06.427690 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz"] Nov 02 01:31:06.434214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:06.434185 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f21c877-59bb5d886f-s4blz"] Nov 02 01:31:08.172147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:08.172029 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5bd001-5b54-453d-908a-0bde329e4100" path="/var/lib/kubelet/pods/ca5bd001-5b54-453d-908a-0bde329e4100/volumes" Nov 02 01:31:10.734105 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.734070 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk"] Nov 02 01:31:10.735255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.735232 4552 cpu_manager.go:414] "containerMap: removing container" podUID="566153ff-2431-4a14-b302-67565cdd2be3" containerName="maas-api" Nov 02 01:31:10.735255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.735253 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566153ff-2431-4a14-b302-67565cdd2be3" containerName="maas-api" Nov 02 01:31:10.735431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.735277 4552 cpu_manager.go:414] "containerMap: removing container" podUID="ca5bd001-5b54-453d-908a-0bde329e4100" containerName="maas-api" Nov 02 01:31:10.735431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.735284 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5bd001-5b54-453d-908a-0bde329e4100" containerName="maas-api" Nov 02 01:31:10.735544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.735529 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="566153ff-2431-4a14-b302-67565cdd2be3" containerName="maas-api" Nov 02 01:31:10.735602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.735551 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5bd001-5b54-453d-908a-0bde329e4100" containerName="maas-api" Nov 02 01:31:10.740942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.740917 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:10.745386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.745362 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9a278165\"" Nov 02 01:31:10.749321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.749291 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk"] Nov 02 01:31:10.893745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.893704 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:10.893961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:10.893759 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tjl\" (UniqueName: \"kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl\") pod \"payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:11.042948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.042857 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:11.043236 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.043209 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tjl\" (UniqueName: \"kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl\") pod \"payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:11.094174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.083764 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:11.176754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.175710 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tjl\" (UniqueName: \"kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl\") pod \"payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:11.336447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.336349 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:31:11.374183 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.374099 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:11.838712 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:31:11.838644 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfcf1c1_d9af_4bf7_b368_85de9875a07d.slice/crio-3dedd1fccf25d1b4cc20db8cbd089a3a2bdc8d25301a6d800157518022730ca5 WatchSource:0}: Error finding container 3dedd1fccf25d1b4cc20db8cbd089a3a2bdc8d25301a6d800157518022730ca5: Status 404 returned error can't find the container with id 3dedd1fccf25d1b4cc20db8cbd089a3a2bdc8d25301a6d800157518022730ca5 Nov 02 01:31:11.840160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:11.840135 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk"] Nov 02 01:31:12.466157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:12.466064 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" event={"ID":"3dfcf1c1-d9af-4bf7-b368-85de9875a07d","Type":"ContainerStarted","Data":"6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5"} Nov 02 01:31:12.466371 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:12.466171 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:12.466371 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:12.466202 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" event={"ID":"3dfcf1c1-d9af-4bf7-b368-85de9875a07d","Type":"ContainerStarted","Data":"3dedd1fccf25d1b4cc20db8cbd089a3a2bdc8d25301a6d800157518022730ca5"} Nov 02 01:31:12.497178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:12.490615 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" podStartSLOduration=2.490592915 podStartE2EDuration="2.490592915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:12.484060293 +0000 UTC m=+972.589849206" watchObservedRunningTime="2026-11-02 01:31:12.490592915 +0000 UTC m=+972.596381829" Nov 02 01:31:14.414151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:14.405821 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6"] Nov 02 01:31:14.417190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:14.415006 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" podUID="97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" containerName="maas-api" containerID="cri-o://2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c" gracePeriod=30 Nov 02 01:31:15.384766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.384690 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:31:15.526427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.526391 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls\") pod \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " Nov 02 01:31:15.526816 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.526470 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs\") pod \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " Nov 02 01:31:15.526816 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.526692 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b\" (UniqueName: \"kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b\") pod \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " Nov 02 01:31:15.526816 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.526750 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca\") pod \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\" (UID: \"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e\") " Nov 02 01:31:15.527204 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.527178 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca" pod "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" (UID: "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:15.527647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.527627 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:15.530048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.530017 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs" pod "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" (UID: "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:15.530228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.530042 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls" pod "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" (UID: "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:15.530228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.530075 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b" pod "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" (UID: "97fdc8aa-dd81-43b2-9984-690f8dd4ef5e"). InnerVolumeSpecName "kube-api-access-kq82b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:15.630079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.630034 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:15.630079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.630077 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:15.630372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.630099 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq82b\" (UniqueName: \"kubernetes.io/projected/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e-kube-api-access-kq82b\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:15.694202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.690190 4552 generic.go:356] "Generic (PLEG): container finished" podID="97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" containerID="2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c" exitCode=0 Nov 02 01:31:15.694202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.690380 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" event={"ID":"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e","Type":"ContainerDied","Data":"2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c"} Nov 02 01:31:15.694202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.690431 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" event={"ID":"97fdc8aa-dd81-43b2-9984-690f8dd4ef5e","Type":"ContainerDied","Data":"7159438309a7ea2dd02940c2b5d21032c7d0ac7904b1f86552c5d0fa7524b793"} Nov 02 01:31:15.694202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.690466 4552 scope.go:122] "RemoveContainer" containerID="2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c" Nov 02 01:31:15.694202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.690659 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6" Nov 02 01:31:15.733862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.733837 4552 scope.go:122] "RemoveContainer" containerID="2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c" Nov 02 01:31:15.734432 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:31:15.734400 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c\": container with ID starting with 2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c not found: ID does not exist" containerID="2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c" Nov 02 01:31:15.734529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.734447 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c"} err="failed to get container status \"2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c\": rpc error: code = NotFound desc = could not find container \"2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c\": container with ID starting with 2ba7f142e26c8875ba092d302d48cbc3c463a5caa10d554b095a42fa842e941c not found: ID does not exist" Nov 02 01:31:15.754384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.754341 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6"] Nov 02 01:31:15.763199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:15.762941 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c45ad5e-c67bd5587-wxzr6"] Nov 02 01:31:16.121184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:16.119883 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" path="/var/lib/kubelet/pods/97fdc8aa-dd81-43b2-9984-690f8dd4ef5e/volumes" Nov 02 01:31:20.521763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.521725 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd"] Nov 02 01:31:20.523827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.523796 4552 cpu_manager.go:414] "containerMap: removing container" podUID="97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" containerName="maas-api" Nov 02 01:31:20.523827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.523826 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" containerName="maas-api" Nov 02 01:31:20.524043 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.524029 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fdc8aa-dd81-43b2-9984-690f8dd4ef5e" containerName="maas-api" Nov 02 01:31:20.530771 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.530745 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.538172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.537983 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd"] Nov 02 01:31:20.538756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.538737 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9a278165-serving-cert\"" Nov 02 01:31:20.586177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.586073 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.586390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.586317 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsg5n\" (UniqueName: \"kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.586461 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.586422 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.586811 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.586596 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.694148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.689767 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.694148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.689919 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.694148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.690016 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.694148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.690239 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsg5n\" (UniqueName: \"kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.694148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.691442 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.718460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.715836 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.723402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.723347 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsg5n\" (UniqueName: \"kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.726965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.726927 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs\") pod \"maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:20.863344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:20.863242 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:21.193481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:21.188283 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd"] Nov 02 01:31:21.749976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:21.749937 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" event={"ID":"b7400488-3581-496b-a1b2-dee90701e3f1","Type":"ContainerStarted","Data":"fcc9c3945edd07daee00d938ed2c4af13fb0f1057db8a135aafa9d4c133a470e"} Nov 02 01:31:22.230138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.227695 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9"] Nov 02 01:31:22.240140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.233989 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.248141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.246492 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-89833f31-serving-cert\"" Nov 02 01:31:22.285139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.275769 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.285139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.275862 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnm6\" (UniqueName: \"kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.285139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.275930 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.285139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.276054 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.295273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.295235 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9"] Nov 02 01:31:22.398188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.397485 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.398188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.397602 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnm6\" (UniqueName: \"kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.398188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.398157 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.398514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.398357 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.407458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.400207 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.426517 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.426481 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.426948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.426925 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.428484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.427360 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnm6\" (UniqueName: \"kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6\") pod \"maas-api-e2e-ait-89833f31-657fd9569d-xlxd9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.613398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.613302 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:22.763046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.763008 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" event={"ID":"b7400488-3581-496b-a1b2-dee90701e3f1","Type":"ContainerStarted","Data":"3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680"} Nov 02 01:31:22.763555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.763106 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:22.783665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.783614 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" podStartSLOduration=2.28868942 podStartE2EDuration="2.783598976s" totalImagesPullingTime="494.909556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:22.782881924 +0000 UTC m=+982.888670838" watchObservedRunningTime="2026-11-02 01:31:22.783598976 +0000 UTC m=+982.889387889" Nov 02 01:31:22.911972 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:31:22.911919 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a11caa7_38ab_4d7c_883a_0a22596870b9.slice/crio-954b48ba56489b4f150209f3697635b396e8c2fdb85b1e755ff009e42faee53a WatchSource:0}: Error finding container 954b48ba56489b4f150209f3697635b396e8c2fdb85b1e755ff009e42faee53a: Status 404 returned error can't find the container with id 954b48ba56489b4f150209f3697635b396e8c2fdb85b1e755ff009e42faee53a Nov 02 01:31:22.916389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:22.916363 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9"] Nov 02 01:31:23.619753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:23.619719 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:23.780188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:23.780042 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" event={"ID":"3a11caa7-38ab-4d7c-883a-0a22596870b9","Type":"ContainerStarted","Data":"1204784ca094d328c4129a55024e18b2fec97c9f5a63c12b4f59a8de2e96e257"} Nov 02 01:31:23.780188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:23.780101 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" event={"ID":"3a11caa7-38ab-4d7c-883a-0a22596870b9","Type":"ContainerStarted","Data":"954b48ba56489b4f150209f3697635b396e8c2fdb85b1e755ff009e42faee53a"} Nov 02 01:31:23.780188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:23.780170 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:23.802165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:23.802087 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" podStartSLOduration=1.239786891 podStartE2EDuration="1.802071081s" totalImagesPullingTime="562.28419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:23.798926007 +0000 UTC m=+983.904714918" watchObservedRunningTime="2026-11-02 01:31:23.802071081 +0000 UTC m=+983.907859996" Nov 02 01:31:27.752250 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:31:27.752198 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:31:28.787813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:28.787782 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:29.796110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:29.796079 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:37.743144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:37.742406 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:37.835140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:37.827328 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:45.741421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:45.741369 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk"] Nov 02 01:31:48.749556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:48.749518 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd"] Nov 02 01:31:48.752228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:48.752177 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" podUID="b7400488-3581-496b-a1b2-dee90701e3f1" containerName="maas-api" containerID="cri-o://3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680" gracePeriod=30 Nov 02 01:31:48.782541 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:48.780699 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" podUID="b7400488-3581-496b-a1b2-dee90701e3f1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.101:8443/health\": dial tcp 10.134.0.101:8443: connect: connection refused" Nov 02 01:31:49.219236 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.219205 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:49.375198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.368753 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs\") pod \"b7400488-3581-496b-a1b2-dee90701e3f1\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " Nov 02 01:31:49.375198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.369149 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca\") pod \"b7400488-3581-496b-a1b2-dee90701e3f1\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " Nov 02 01:31:49.375198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.369252 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls\") pod \"b7400488-3581-496b-a1b2-dee90701e3f1\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " Nov 02 01:31:49.375198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.369470 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n\" (UniqueName: \"kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n\") pod \"b7400488-3581-496b-a1b2-dee90701e3f1\" (UID: \"b7400488-3581-496b-a1b2-dee90701e3f1\") " Nov 02 01:31:49.389299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.383682 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca" pod "b7400488-3581-496b-a1b2-dee90701e3f1" (UID: "b7400488-3581-496b-a1b2-dee90701e3f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:49.396913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.396625 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls" pod "b7400488-3581-496b-a1b2-dee90701e3f1" (UID: "b7400488-3581-496b-a1b2-dee90701e3f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:49.419146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.412857 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs" pod "b7400488-3581-496b-a1b2-dee90701e3f1" (UID: "b7400488-3581-496b-a1b2-dee90701e3f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:49.438061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.437973 4552 generic.go:356] "Generic (PLEG): container finished" podID="b7400488-3581-496b-a1b2-dee90701e3f1" containerID="3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680" exitCode=0 Nov 02 01:31:49.438288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.438091 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" event={"ID":"b7400488-3581-496b-a1b2-dee90701e3f1","Type":"ContainerDied","Data":"3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680"} Nov 02 01:31:49.438288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.438166 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" event={"ID":"b7400488-3581-496b-a1b2-dee90701e3f1","Type":"ContainerDied","Data":"fcc9c3945edd07daee00d938ed2c4af13fb0f1057db8a135aafa9d4c133a470e"} Nov 02 01:31:49.438288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.438197 4552 scope.go:122] "RemoveContainer" containerID="3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680" Nov 02 01:31:49.438462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.438374 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd" Nov 02 01:31:49.474213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.471656 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:49.474213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.471690 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7400488-3581-496b-a1b2-dee90701e3f1-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:49.474213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.471705 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7400488-3581-496b-a1b2-dee90701e3f1-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:49.488811 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.488755 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n" pod "b7400488-3581-496b-a1b2-dee90701e3f1" (UID: "b7400488-3581-496b-a1b2-dee90701e3f1"). InnerVolumeSpecName "kube-api-access-wsg5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:49.517242 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.517216 4552 scope.go:122] "RemoveContainer" containerID="3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680" Nov 02 01:31:49.517729 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:31:49.517625 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680\": container with ID starting with 3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680 not found: ID does not exist" containerID="3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680" Nov 02 01:31:49.517729 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.517667 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680"} err="failed to get container status \"3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680\": rpc error: code = NotFound desc = could not find container \"3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680\": container with ID starting with 3670ccec0a4c2428fad4f55dc73982c8209480ba9f188a97c67ff63df4344680 not found: ID does not exist" Nov 02 01:31:49.577955 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.577912 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsg5n\" (UniqueName: \"kubernetes.io/projected/b7400488-3581-496b-a1b2-dee90701e3f1-kube-api-access-wsg5n\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:49.789239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.789202 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd"] Nov 02 01:31:49.796289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:49.796258 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9a278165-9775dc88f-vqmsd"] Nov 02 01:31:50.049569 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.049474 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7400488-3581-496b-a1b2-dee90701e3f1" path="/var/lib/kubelet/pods/b7400488-3581-496b-a1b2-dee90701e3f1/volumes" Nov 02 01:31:50.288752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.288719 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9"] Nov 02 01:31:50.290019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.289488 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" podUID="3a11caa7-38ab-4d7c-883a-0a22596870b9" containerName="maas-api" containerID="cri-o://1204784ca094d328c4129a55024e18b2fec97c9f5a63c12b4f59a8de2e96e257" gracePeriod=30 Nov 02 01:31:50.450079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.450044 4552 generic.go:356] "Generic (PLEG): container finished" podID="3a11caa7-38ab-4d7c-883a-0a22596870b9" containerID="1204784ca094d328c4129a55024e18b2fec97c9f5a63c12b4f59a8de2e96e257" exitCode=0 Nov 02 01:31:50.450280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.450131 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" event={"ID":"3a11caa7-38ab-4d7c-883a-0a22596870b9","Type":"ContainerDied","Data":"1204784ca094d328c4129a55024e18b2fec97c9f5a63c12b4f59a8de2e96e257"} Nov 02 01:31:50.778363 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.778316 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" podUID="3dfcf1c1-d9af-4bf7-b368-85de9875a07d" containerName="payload-processing" containerID="cri-o://6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5" gracePeriod=25 Nov 02 01:31:50.825392 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.825361 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:50.942766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.942731 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls\") pod \"3a11caa7-38ab-4d7c-883a-0a22596870b9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " Nov 02 01:31:50.942917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.942848 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca\") pod \"3a11caa7-38ab-4d7c-883a-0a22596870b9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " Nov 02 01:31:50.943039 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.943026 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6\" (UniqueName: \"kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6\") pod \"3a11caa7-38ab-4d7c-883a-0a22596870b9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " Nov 02 01:31:50.943086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.943066 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs\") pod \"3a11caa7-38ab-4d7c-883a-0a22596870b9\" (UID: \"3a11caa7-38ab-4d7c-883a-0a22596870b9\") " Nov 02 01:31:50.943332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.943304 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca" pod "3a11caa7-38ab-4d7c-883a-0a22596870b9" (UID: "3a11caa7-38ab-4d7c-883a-0a22596870b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:50.943915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.943863 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a11caa7-38ab-4d7c-883a-0a22596870b9-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:50.945631 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.945604 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs" pod "3a11caa7-38ab-4d7c-883a-0a22596870b9" (UID: "3a11caa7-38ab-4d7c-883a-0a22596870b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:50.945969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.945942 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls" pod "3a11caa7-38ab-4d7c-883a-0a22596870b9" (UID: "3a11caa7-38ab-4d7c-883a-0a22596870b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:31:50.946056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:50.946001 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6" pod "3a11caa7-38ab-4d7c-883a-0a22596870b9" (UID: "3a11caa7-38ab-4d7c-883a-0a22596870b9"). InnerVolumeSpecName "kube-api-access-hdnm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:51.046420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.045242 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnm6\" (UniqueName: \"kubernetes.io/projected/3a11caa7-38ab-4d7c-883a-0a22596870b9-kube-api-access-hdnm6\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:51.046420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.045269 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:51.046420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.045278 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a11caa7-38ab-4d7c-883a-0a22596870b9-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:51.202683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.202316 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:51.254657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.254622 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl\" (UniqueName: \"kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl\") pod \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " Nov 02 01:31:51.259025 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.258971 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume\") pod \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\" (UID: \"3dfcf1c1-d9af-4bf7-b368-85de9875a07d\") " Nov 02 01:31:51.260286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.260258 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume" pod "3dfcf1c1-d9af-4bf7-b368-85de9875a07d" (UID: "3dfcf1c1-d9af-4bf7-b368-85de9875a07d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:51.263056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.263014 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl" pod "3dfcf1c1-d9af-4bf7-b368-85de9875a07d" (UID: "3dfcf1c1-d9af-4bf7-b368-85de9875a07d"). InnerVolumeSpecName "kube-api-access-k5tjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:51.361405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.361313 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:51.361405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.361342 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tjl\" (UniqueName: \"kubernetes.io/projected/3dfcf1c1-d9af-4bf7-b368-85de9875a07d-kube-api-access-k5tjl\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:51.460636 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.460607 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" Nov 02 01:31:51.460636 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.460612 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9" event={"ID":"3a11caa7-38ab-4d7c-883a-0a22596870b9","Type":"ContainerDied","Data":"954b48ba56489b4f150209f3697635b396e8c2fdb85b1e755ff009e42faee53a"} Nov 02 01:31:51.460883 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.460680 4552 scope.go:122] "RemoveContainer" containerID="1204784ca094d328c4129a55024e18b2fec97c9f5a63c12b4f59a8de2e96e257" Nov 02 01:31:51.462637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.462617 4552 generic.go:356] "Generic (PLEG): container finished" podID="3dfcf1c1-d9af-4bf7-b368-85de9875a07d" containerID="6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5" exitCode=0 Nov 02 01:31:51.462744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.462693 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" event={"ID":"3dfcf1c1-d9af-4bf7-b368-85de9875a07d","Type":"ContainerDied","Data":"6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5"} Nov 02 01:31:51.462744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.462728 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" event={"ID":"3dfcf1c1-d9af-4bf7-b368-85de9875a07d","Type":"ContainerDied","Data":"3dedd1fccf25d1b4cc20db8cbd089a3a2bdc8d25301a6d800157518022730ca5"} Nov 02 01:31:51.462863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.462827 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk" Nov 02 01:31:51.473853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.473836 4552 scope.go:122] "RemoveContainer" containerID="6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5" Nov 02 01:31:51.483896 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.483877 4552 scope.go:122] "RemoveContainer" containerID="6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5" Nov 02 01:31:51.484207 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:31:51.484186 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5\": container with ID starting with 6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5 not found: ID does not exist" containerID="6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5" Nov 02 01:31:51.484271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.484221 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5"} err="failed to get container status \"6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5\": rpc error: code = NotFound desc = could not find container \"6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5\": container with ID starting with 6093d9f4bf40b10deb47c6f51d332a720dbe84b48ae8d94c9bb6d2d0f55599e5 not found: ID does not exist" Nov 02 01:31:51.486904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.486882 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9"] Nov 02 01:31:51.492618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.492598 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89833f31-657fd9569d-xlxd9"] Nov 02 01:31:51.505982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.505959 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk"] Nov 02 01:31:51.510253 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:51.510234 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9a278165-84d559b847-6rvxk"] Nov 02 01:31:52.055797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:52.055756 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a11caa7-38ab-4d7c-883a-0a22596870b9" path="/var/lib/kubelet/pods/3a11caa7-38ab-4d7c-883a-0a22596870b9/volumes" Nov 02 01:31:52.056420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:52.056402 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfcf1c1-d9af-4bf7-b368-85de9875a07d" path="/var/lib/kubelet/pods/3dfcf1c1-d9af-4bf7-b368-85de9875a07d/volumes" Nov 02 01:31:57.754835 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:31:57.754791 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:31:57.860073 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.860035 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4"] Nov 02 01:31:57.861293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861266 4552 cpu_manager.go:414] "containerMap: removing container" podUID="3dfcf1c1-d9af-4bf7-b368-85de9875a07d" containerName="payload-processing" Nov 02 01:31:57.861293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861293 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfcf1c1-d9af-4bf7-b368-85de9875a07d" containerName="payload-processing" Nov 02 01:31:57.861486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861325 4552 cpu_manager.go:414] "containerMap: removing container" podUID="b7400488-3581-496b-a1b2-dee90701e3f1" containerName="maas-api" Nov 02 01:31:57.861486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861333 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7400488-3581-496b-a1b2-dee90701e3f1" containerName="maas-api" Nov 02 01:31:57.861486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861393 4552 cpu_manager.go:414] "containerMap: removing container" podUID="3a11caa7-38ab-4d7c-883a-0a22596870b9" containerName="maas-api" Nov 02 01:31:57.861486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861400 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a11caa7-38ab-4d7c-883a-0a22596870b9" containerName="maas-api" Nov 02 01:31:57.861678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861578 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a11caa7-38ab-4d7c-883a-0a22596870b9" containerName="maas-api" Nov 02 01:31:57.861678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861594 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfcf1c1-d9af-4bf7-b368-85de9875a07d" containerName="payload-processing" Nov 02 01:31:57.861678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.861608 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7400488-3581-496b-a1b2-dee90701e3f1" containerName="maas-api" Nov 02 01:31:57.866375 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.866348 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:57.873268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.873242 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-eeeb1a\"" Nov 02 01:31:57.873504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.873478 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4"] Nov 02 01:31:57.923532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.923478 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:57.923744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.923636 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rcx\" (UniqueName: \"kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx\") pod \"payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:57.946448 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.946408 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr"] Nov 02 01:31:57.951248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.951221 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:57.955264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.955232 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-eeeb1a-dockercfg-v84hw\"" Nov 02 01:31:57.960721 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:57.960689 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr"] Nov 02 01:31:58.115193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.114811 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:58.115193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.114935 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.115193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.114969 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rcx\" (UniqueName: \"kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx\") pod \"payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:58.115193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.115060 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkxp\" (UniqueName: \"kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp\") pod \"payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.137233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.115979 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:58.229140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.228769 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.229140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.228935 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkxp\" (UniqueName: \"kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp\") pod \"payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.232146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.230457 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.266829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.266793 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rcx\" (UniqueName: \"kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx\") pod \"payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:58.370185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.369692 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkxp\" (UniqueName: \"kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp\") pod \"payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.410365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.410332 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:58.496661 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.496640 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:58.720194 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:31:58.720157 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03985cd_4dd2_456e_9683_a4de3c0fa09c.slice/crio-59281b3382e65c94935ded17856bd5f30674fee7715f2cf75e5a6ff7db01e7f9 WatchSource:0}: Error finding container 59281b3382e65c94935ded17856bd5f30674fee7715f2cf75e5a6ff7db01e7f9: Status 404 returned error can't find the container with id 59281b3382e65c94935ded17856bd5f30674fee7715f2cf75e5a6ff7db01e7f9 Nov 02 01:31:58.723579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.723545 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr"] Nov 02 01:31:58.900580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:58.900540 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4"] Nov 02 01:31:58.901176 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:31:58.901102 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72601e1_f811_4ae5_bb74_af8ced4a5363.slice/crio-68f13491cba80d926f49ef26cfe066c7c31bf149aaa8df0adb6d26fda2f32e31 WatchSource:0}: Error finding container 68f13491cba80d926f49ef26cfe066c7c31bf149aaa8df0adb6d26fda2f32e31: Status 404 returned error can't find the container with id 68f13491cba80d926f49ef26cfe066c7c31bf149aaa8df0adb6d26fda2f32e31 Nov 02 01:31:59.590372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.590335 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" event={"ID":"d03985cd-4dd2-456e-9683-a4de3c0fa09c","Type":"ContainerStarted","Data":"d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535"} Nov 02 01:31:59.590671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.590651 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" event={"ID":"d03985cd-4dd2-456e-9683-a4de3c0fa09c","Type":"ContainerStarted","Data":"59281b3382e65c94935ded17856bd5f30674fee7715f2cf75e5a6ff7db01e7f9"} Nov 02 01:31:59.591911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.591888 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:31:59.599744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.599103 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" event={"ID":"d72601e1-f811-4ae5-bb74-af8ced4a5363","Type":"ContainerStarted","Data":"0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772"} Nov 02 01:31:59.599744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.599188 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" event={"ID":"d72601e1-f811-4ae5-bb74-af8ced4a5363","Type":"ContainerStarted","Data":"68f13491cba80d926f49ef26cfe066c7c31bf149aaa8df0adb6d26fda2f32e31"} Nov 02 01:31:59.600307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.600280 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:31:59.613682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.613620 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" podStartSLOduration=2.613601574 podStartE2EDuration="2.613601574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:59.612359418 +0000 UTC m=+1019.718148332" watchObservedRunningTime="2026-11-02 01:31:59.613601574 +0000 UTC m=+1019.719390489" Nov 02 01:31:59.640543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:31:59.640461 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" podStartSLOduration=2.640442297 podStartE2EDuration="2.640442297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:59.634207248 +0000 UTC m=+1019.739996175" watchObservedRunningTime="2026-11-02 01:31:59.640442297 +0000 UTC m=+1019.746231212" Nov 02 01:32:03.034175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.030775 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl"] Nov 02 01:32:03.053515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.050044 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.053515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.053249 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl"] Nov 02 01:32:03.063185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.057313 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-eeeb1a-serving-cert\"" Nov 02 01:32:03.239404 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.239354 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.239615 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.239545 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.239706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.239687 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.239804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.239785 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4jf\" (UniqueName: \"kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.341826 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.341707 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.341826 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.341817 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4jf\" (UniqueName: \"kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.342088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.342071 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.342243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.342225 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.342991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.342964 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.344716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.344678 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.344818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.344757 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.354315 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.354288 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4jf\" (UniqueName: \"kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf\") pod \"maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.378542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.378503 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:03.800064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:03.799848 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl"] Nov 02 01:32:04.748214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:04.748154 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" event={"ID":"946026f3-a365-45bd-9c13-98c484fd7080","Type":"ContainerStarted","Data":"110b25dbeb2983e5c2e09e0003b5bc4938a5032b6d7f1ce21f0a1819a5e4a435"} Nov 02 01:32:04.748822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:04.748228 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" event={"ID":"946026f3-a365-45bd-9c13-98c484fd7080","Type":"ContainerStarted","Data":"ddea626a04badab9d64de3c96625155355f2c309644be29fa337ad76802d2002"} Nov 02 01:32:04.748959 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:04.748821 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:04.773685 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:04.773624 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" podStartSLOduration=1.2195344989999999 podStartE2EDuration="1.773605302s" totalImagesPullingTime="554.070803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:04.76832889 +0000 UTC m=+1024.874117806" watchObservedRunningTime="2026-11-02 01:32:04.773605302 +0000 UTC m=+1024.879394215" Nov 02 01:32:07.039660 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:07.039625 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:32:10.797355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:10.793317 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:32:10.832240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:10.832182 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 02 01:32:11.641511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:11.641476 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:32:11.770139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:11.769191 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:32:27.805451 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:32:27.805412 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:32:29.037366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.037322 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9"] Nov 02 01:32:29.043491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.043464 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.052950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.050533 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-c22a0d-serving-cert\"" Nov 02 01:32:29.053166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.053140 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9"] Nov 02 01:32:29.127135 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.127085 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpwz\" (UniqueName: \"kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.127317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.127208 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.127381 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.127323 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.127516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.127499 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.228936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.228894 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.229186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.229167 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpwz\" (UniqueName: \"kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.229272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.229207 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.229336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.229267 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.230020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.229938 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.231607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.231565 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.231834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.231812 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.257665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.257627 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpwz\" (UniqueName: \"kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz\") pod \"maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.382003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.380426 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:29.800287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.800245 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9"] Nov 02 01:32:29.983416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:29.983383 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" event={"ID":"a193d88f-a4a0-4def-95d8-2087cbc94d02","Type":"ContainerStarted","Data":"8e332234dd121ce25d80637b7647d08bde7c279f9792641d0c6e71a7d0ac58d3"} Nov 02 01:32:31.000641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:31.000599 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" event={"ID":"a193d88f-a4a0-4def-95d8-2087cbc94d02","Type":"ContainerStarted","Data":"774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e"} Nov 02 01:32:31.001084 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:31.000736 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:31.025266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:31.025213 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" podStartSLOduration=1.4904022860000001 podStartE2EDuration="2.025198149s" totalImagesPullingTime="534.795863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:31.020490606 +0000 UTC m=+1051.126279522" watchObservedRunningTime="2026-11-02 01:32:31.025198149 +0000 UTC m=+1051.130987062" Nov 02 01:32:37.014517 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:37.014484 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:32:48.821321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.821284 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq"] Nov 02 01:32:48.827664 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.827630 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:48.847353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.847290 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50537f4f-serving-cert\"" Nov 02 01:32:48.856520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.856471 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq"] Nov 02 01:32:48.922693 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.922659 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:48.922958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.922940 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kb5c\" (UniqueName: \"kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:48.923082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.923067 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:48.923178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:48.923097 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.035141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.030456 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kb5c\" (UniqueName: \"kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.035141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.030648 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.035141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.030680 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.035141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.030779 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.052138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.051501 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.063140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.054877 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kb5c\" (UniqueName: \"kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.094194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.090416 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.094194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.092288 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.161514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.161483 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:49.620907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.614224 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq"] Nov 02 01:32:49.636570 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:32:49.635414 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704acddf_addc_4042_bdd4_a69edcc710ff.slice/crio-29f49faa4119dd574fb5b3c9c1b4d3f7b6d37f56b9201c8774506ba46f67c1cb WatchSource:0}: Error finding container 29f49faa4119dd574fb5b3c9c1b4d3f7b6d37f56b9201c8774506ba46f67c1cb: Status 404 returned error can't find the container with id 29f49faa4119dd574fb5b3c9c1b4d3f7b6d37f56b9201c8774506ba46f67c1cb Nov 02 01:32:49.653168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.650401 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb"] Nov 02 01:32:49.653168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:49.650683 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" podUID="2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" containerName="maas-api" containerID="cri-o://7222dfc5ec04d33a42ac680044c9d1435234f1ff7e5c8c694d987583cea715ec" gracePeriod=30 Nov 02 01:32:50.259081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.259045 4552 generic.go:356] "Generic (PLEG): container finished" podID="2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" containerID="7222dfc5ec04d33a42ac680044c9d1435234f1ff7e5c8c694d987583cea715ec" exitCode=0 Nov 02 01:32:50.275165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.275104 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" event={"ID":"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd","Type":"ContainerDied","Data":"7222dfc5ec04d33a42ac680044c9d1435234f1ff7e5c8c694d987583cea715ec"} Nov 02 01:32:50.275360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.275190 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" event={"ID":"704acddf-addc-4042-bdd4-a69edcc710ff","Type":"ContainerStarted","Data":"29f49faa4119dd574fb5b3c9c1b4d3f7b6d37f56b9201c8774506ba46f67c1cb"} Nov 02 01:32:50.373562 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.373464 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:32:50.477777 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.477740 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls\") pod \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " Nov 02 01:32:50.477984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.477818 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs\") pod \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " Nov 02 01:32:50.477984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.477888 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca\") pod \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " Nov 02 01:32:50.477984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.477954 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh\" (UniqueName: \"kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh\") pod \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\" (UID: \"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd\") " Nov 02 01:32:50.478832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.478347 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca" pod "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" (UID: "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:32:50.478958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.478924 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:32:50.499210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.499150 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls" pod "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" (UID: "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:32:50.499672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.499504 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh" pod "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" (UID: "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd"). InnerVolumeSpecName "kube-api-access-4frhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:32:50.505185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.504762 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs" pod "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" (UID: "2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:32:50.580405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.580324 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:32:50.580405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.580353 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:32:50.580405 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:50.580365 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frhh\" (UniqueName: \"kubernetes.io/projected/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd-kube-api-access-4frhh\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:32:51.282485 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.282451 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" Nov 02 01:32:51.282962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.282483 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb" event={"ID":"2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd","Type":"ContainerDied","Data":"666f16ab95e9592694635dccc7f3467a181da08a1d9a5fde86e27f12e3ac7e0c"} Nov 02 01:32:51.282962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.282564 4552 scope.go:122] "RemoveContainer" containerID="7222dfc5ec04d33a42ac680044c9d1435234f1ff7e5c8c694d987583cea715ec" Nov 02 01:32:51.284969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.284940 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" event={"ID":"704acddf-addc-4042-bdd4-a69edcc710ff","Type":"ContainerStarted","Data":"c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0"} Nov 02 01:32:51.285052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.285012 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:51.314904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.314812 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" podStartSLOduration=2.589800553 podStartE2EDuration="3.314797151s" totalImagesPullingTime="724.996598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:51.311260809 +0000 UTC m=+1071.417049723" watchObservedRunningTime="2026-11-02 01:32:51.314797151 +0000 UTC m=+1071.420586064" Nov 02 01:32:51.329059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.329028 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb"] Nov 02 01:32:51.333412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:51.333389 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-616e3ed7-5b967d7b6c-wvbmb"] Nov 02 01:32:52.046944 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:52.046900 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" path="/var/lib/kubelet/pods/2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd/volumes" Nov 02 01:32:57.301910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:32:57.301879 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:32:57.894642 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:32:57.893850 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:33:03.097679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.097649 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74"] Nov 02 01:33:03.099010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.098991 4552 cpu_manager.go:414] "containerMap: removing container" podUID="2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" containerName="maas-api" Nov 02 01:33:03.099125 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.099101 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" containerName="maas-api" Nov 02 01:33:03.099394 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.099382 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1b1233-9f7b-49b3-bd62-a8d4dce02dbd" containerName="maas-api" Nov 02 01:33:03.107151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.107130 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.116270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.116105 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-50537f4f\"/\"kube-root-ca.crt\"" Nov 02 01:33:03.121815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.121789 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74"] Nov 02 01:33:03.122438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.122411 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.122512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.122501 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.122614 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.122600 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.122742 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.122727 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.122812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.122797 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrc5\" (UniqueName: \"kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.122964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.122948 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.125954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.125930 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-50537f4f\"/\"openshift-service-ca.crt\"" Nov 02 01:33:03.126160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.126100 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-50537f4f\"/\"e2e-lifecycle-model-50537f4f-kserve-self-signed-certs\"" Nov 02 01:33:03.126216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.126196 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-50537f4f\"/\"default-dockercfg-gkqpq\"" Nov 02 01:33:03.224507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224473 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.224687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224532 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.224687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224571 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.224687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224651 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.224845 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224705 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrc5\" (UniqueName: \"kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.224845 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224807 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.224950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.224873 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.225086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.225063 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.225231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.225209 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.226969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.226952 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.227375 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.227354 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.246145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.238672 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrc5\" (UniqueName: \"kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5\") pod \"e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.434815 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.433529 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:03.799426 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:03.799396 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74"] Nov 02 01:33:03.801567 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:33:03.801543 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc901fb0_87d2_44b1_9745_39bf870271d3.slice/crio-88567f2028e52f683857581023ebed9cdcf0dd15e994f9297ccfc3764680489d WatchSource:0}: Error finding container 88567f2028e52f683857581023ebed9cdcf0dd15e994f9297ccfc3764680489d: Status 404 returned error can't find the container with id 88567f2028e52f683857581023ebed9cdcf0dd15e994f9297ccfc3764680489d Nov 02 01:33:04.450868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:04.450823 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" event={"ID":"cc901fb0-87d2-44b1-9745-39bf870271d3","Type":"ContainerStarted","Data":"cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f"} Nov 02 01:33:04.451294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:04.450881 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" event={"ID":"cc901fb0-87d2-44b1-9745-39bf870271d3","Type":"ContainerStarted","Data":"88567f2028e52f683857581023ebed9cdcf0dd15e994f9297ccfc3764680489d"} Nov 02 01:33:04.473669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:04.473617 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" podStartSLOduration=1.473604689 podStartE2EDuration="1.473604689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:04.471354359 +0000 UTC m=+1084.577143278" watchObservedRunningTime="2026-11-02 01:33:04.473604689 +0000 UTC m=+1084.579393602" Nov 02 01:33:05.435387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:05.435351 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:08.867596 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:08.867558 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:33:13.487776 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:13.487738 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:13.728973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:13.728918 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.831082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:13.830966 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.453342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:14.453310 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:14.746543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:14.746453 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:33:21.158767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.158735 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz"] Nov 02 01:33:21.163802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.163779 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.192137 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.187023 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2ebe99-serving-cert\"" Nov 02 01:33:21.206986 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.197437 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz"] Nov 02 01:33:21.251336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.251281 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.251592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.251505 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9sqw\" (UniqueName: \"kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.251592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.251541 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.251758 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.251708 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.372189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.365422 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9sqw\" (UniqueName: \"kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.372509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.372483 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.375202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.373401 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.387220 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:33:21.375474 4552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-serving-cert: secret "maas-api-e2e-api-a-2ebe99-serving-cert" not found Nov 02 01:33:21.387535 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:33:21.387518 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls podName:d3ac615d-96bb-477f-9486-10052cf8f4f0 nodeName:}" failed. No retries permitted until 2026-11-02 01:33:21.887487396 +0000 UTC m=+1101.993276291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls") pod "maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" (UID: "d3ac615d-96bb-477f-9486-10052cf8f4f0") : secret "maas-api-e2e-api-a-2ebe99-serving-cert" not found Nov 02 01:33:21.387891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.373601 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.388262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.388241 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.409882 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.409807 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.421369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.421336 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9sqw\" (UniqueName: \"kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.925238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.925205 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:21.928406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:21.928377 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:22.107096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:22.107067 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:22.409754 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:33:22.409709 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ac615d_96bb_477f_9486_10052cf8f4f0.slice/crio-c5465ed3d29d0e303aa1009270fb945cb37b7db917e72b11b4b2a43f887b97ac WatchSource:0}: Error finding container c5465ed3d29d0e303aa1009270fb945cb37b7db917e72b11b4b2a43f887b97ac: Status 404 returned error can't find the container with id c5465ed3d29d0e303aa1009270fb945cb37b7db917e72b11b4b2a43f887b97ac Nov 02 01:33:22.410207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:22.410080 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz"] Nov 02 01:33:23.357640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:23.357584 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" event={"ID":"d3ac615d-96bb-477f-9486-10052cf8f4f0","Type":"ContainerStarted","Data":"25301bb89ee60a5e3ab4ce99ed5b44db1fc6e51aa9d1020f4606f4c678df3d90"} Nov 02 01:33:23.357927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:23.357690 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" event={"ID":"d3ac615d-96bb-477f-9486-10052cf8f4f0","Type":"ContainerStarted","Data":"c5465ed3d29d0e303aa1009270fb945cb37b7db917e72b11b4b2a43f887b97ac"} Nov 02 01:33:23.357927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:23.357729 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:23.380704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:23.380636 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" podStartSLOduration=1.895270482 podStartE2EDuration="2.380617519s" totalImagesPullingTime="485.347037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:23.377811116 +0000 UTC m=+1103.483600031" watchObservedRunningTime="2026-11-02 01:33:23.380617519 +0000 UTC m=+1103.486406434" Nov 02 01:33:26.524193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:26.524161 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:33:27.800378 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:33:27.800338 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:33:29.435061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:29.434775 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:33:30.649283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:30.649238 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:30.707250 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:30.707197 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 01:33:30.891144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:30.890266 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:31.236545 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:31.236494 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:49.021680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.021594 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6"] Nov 02 01:33:49.026479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.026457 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.037879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.037849 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6"] Nov 02 01:33:49.046395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.046370 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-37b78d-serving-cert\"" Nov 02 01:33:49.202199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.200572 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dgw\" (UniqueName: \"kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.202199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.200651 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.202199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.200898 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.202199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.201072 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.307338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.307244 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.312877 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.312761 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.313066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.313013 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dgw\" (UniqueName: \"kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.313206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.313097 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.317481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.309015 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.326809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.325136 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.326809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.326597 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.335545 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.335515 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dgw\" (UniqueName: \"kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw\") pod \"maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.363223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.363193 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:49.661013 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:33:49.660970 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664684ad_781b_4763_9b67_d1506dac2664.slice/crio-15897306ace0bc98ef4e28d778d5982452b312b78794f4d7a9b8f970a08cc2ad WatchSource:0}: Error finding container 15897306ace0bc98ef4e28d778d5982452b312b78794f4d7a9b8f970a08cc2ad: Status 404 returned error can't find the container with id 15897306ace0bc98ef4e28d778d5982452b312b78794f4d7a9b8f970a08cc2ad Nov 02 01:33:49.661502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.661479 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6"] Nov 02 01:33:49.956305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:49.956197 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" event={"ID":"664684ad-781b-4763-9b67-d1506dac2664","Type":"ContainerStarted","Data":"15897306ace0bc98ef4e28d778d5982452b312b78794f4d7a9b8f970a08cc2ad"} Nov 02 01:33:50.992853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:50.992105 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" event={"ID":"664684ad-781b-4763-9b67-d1506dac2664","Type":"ContainerStarted","Data":"8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9"} Nov 02 01:33:50.992853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:50.992350 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:51.013668 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:51.013609 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" podStartSLOduration=2.407953532 podStartE2EDuration="3.013594155s" totalImagesPullingTime="605.640623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:51.011703597 +0000 UTC m=+1131.117492529" watchObservedRunningTime="2026-11-02 01:33:51.013594155 +0000 UTC m=+1131.119383069" Nov 02 01:33:57.007336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:33:57.007306 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:33:57.759249 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:33:57.759207 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:34:03.932099 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:03.932069 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:34:27.796249 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:34:27.794386 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:34:37.602812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:37.602770 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz"] Nov 02 01:34:37.603318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:37.603028 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" podUID="d3ac615d-96bb-477f-9486-10052cf8f4f0" containerName="maas-api" containerID="cri-o://25301bb89ee60a5e3ab4ce99ed5b44db1fc6e51aa9d1020f4606f4c678df3d90" gracePeriod=30 Nov 02 01:34:38.178089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.178050 4552 generic.go:356] "Generic (PLEG): container finished" podID="d3ac615d-96bb-477f-9486-10052cf8f4f0" containerID="25301bb89ee60a5e3ab4ce99ed5b44db1fc6e51aa9d1020f4606f4c678df3d90" exitCode=0 Nov 02 01:34:38.183254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.183166 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" event={"ID":"d3ac615d-96bb-477f-9486-10052cf8f4f0","Type":"ContainerDied","Data":"25301bb89ee60a5e3ab4ce99ed5b44db1fc6e51aa9d1020f4606f4c678df3d90"} Nov 02 01:34:38.287320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.287291 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:34:38.391575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.391534 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca\") pod \"d3ac615d-96bb-477f-9486-10052cf8f4f0\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " Nov 02 01:34:38.391775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.391690 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw\" (UniqueName: \"kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw\") pod \"d3ac615d-96bb-477f-9486-10052cf8f4f0\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " Nov 02 01:34:38.391845 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.391815 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs\") pod \"d3ac615d-96bb-477f-9486-10052cf8f4f0\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " Nov 02 01:34:38.391905 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.391854 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca" pod "d3ac615d-96bb-477f-9486-10052cf8f4f0" (UID: "d3ac615d-96bb-477f-9486-10052cf8f4f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:34:38.391959 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.391908 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\") pod \"d3ac615d-96bb-477f-9486-10052cf8f4f0\" (UID: \"d3ac615d-96bb-477f-9486-10052cf8f4f0\") " Nov 02 01:34:38.392585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.392561 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ac615d-96bb-477f-9486-10052cf8f4f0-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:34:38.394180 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.394152 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs" pod "d3ac615d-96bb-477f-9486-10052cf8f4f0" (UID: "d3ac615d-96bb-477f-9486-10052cf8f4f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:34:38.394272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.394201 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls" pod "d3ac615d-96bb-477f-9486-10052cf8f4f0" (UID: "d3ac615d-96bb-477f-9486-10052cf8f4f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:34:38.394272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.394215 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw" pod "d3ac615d-96bb-477f-9486-10052cf8f4f0" (UID: "d3ac615d-96bb-477f-9486-10052cf8f4f0"). InnerVolumeSpecName "kube-api-access-d9sqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:34:38.501845 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.501804 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9sqw\" (UniqueName: \"kubernetes.io/projected/d3ac615d-96bb-477f-9486-10052cf8f4f0-kube-api-access-d9sqw\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:34:38.503580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.503436 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:34:38.503740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:38.503588 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ac615d-96bb-477f-9486-10052cf8f4f0-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:34:39.203651 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:39.203612 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" Nov 02 01:34:39.204193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:39.203635 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz" event={"ID":"d3ac615d-96bb-477f-9486-10052cf8f4f0","Type":"ContainerDied","Data":"c5465ed3d29d0e303aa1009270fb945cb37b7db917e72b11b4b2a43f887b97ac"} Nov 02 01:34:39.204193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:39.203731 4552 scope.go:122] "RemoveContainer" containerID="25301bb89ee60a5e3ab4ce99ed5b44db1fc6e51aa9d1020f4606f4c678df3d90" Nov 02 01:34:39.233200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:39.233171 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz"] Nov 02 01:34:39.243414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:39.243381 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ebe99-cbdc565ff-phmvz"] Nov 02 01:34:40.063095 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:40.063052 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ac615d-96bb-477f-9486-10052cf8f4f0" path="/var/lib/kubelet/pods/d3ac615d-96bb-477f-9486-10052cf8f4f0/volumes" Nov 02 01:34:54.781026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.780993 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6589-d2cedc"] Nov 02 01:34:54.781846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.781829 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d3ac615d-96bb-477f-9486-10052cf8f4f0" containerName="maas-api" Nov 02 01:34:54.781899 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.781846 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ac615d-96bb-477f-9486-10052cf8f4f0" containerName="maas-api" Nov 02 01:34:54.781979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.781970 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ac615d-96bb-477f-9486-10052cf8f4f0" containerName="maas-api" Nov 02 01:34:54.786042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.786017 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:34:54.792192 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.792167 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6589-d2cedc"] Nov 02 01:34:54.814790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.814758 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vhtnz\"" Nov 02 01:34:54.933309 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:54.933270 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzrs\" (UniqueName: \"kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs\") pod \"test-curl-6589-d2cedc\" (UID: \"52cc646b-5758-438f-a0ac-97d20025b54c\") " pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:34:55.035232 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:55.035142 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzrs\" (UniqueName: \"kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs\") pod \"test-curl-6589-d2cedc\" (UID: \"52cc646b-5758-438f-a0ac-97d20025b54c\") " pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:34:55.048010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:55.047974 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzrs\" (UniqueName: \"kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs\") pod \"test-curl-6589-d2cedc\" (UID: \"52cc646b-5758-438f-a0ac-97d20025b54c\") " pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:34:55.102322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:55.102279 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:34:55.390839 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:34:55.390807 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52cc646b_5758_438f_a0ac_97d20025b54c.slice/crio-c0ae14ffc157f714265fce1009b56c69f39e284e0eb66f09727a8fac7073c21f WatchSource:0}: Error finding container c0ae14ffc157f714265fce1009b56c69f39e284e0eb66f09727a8fac7073c21f: Status 404 returned error can't find the container with id c0ae14ffc157f714265fce1009b56c69f39e284e0eb66f09727a8fac7073c21f Nov 02 01:34:55.392418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:55.392393 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6589-d2cedc"] Nov 02 01:34:55.599705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:55.599667 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6589-d2cedc" event={"ID":"52cc646b-5758-438f-a0ac-97d20025b54c","Type":"ContainerStarted","Data":"c0ae14ffc157f714265fce1009b56c69f39e284e0eb66f09727a8fac7073c21f"} Nov 02 01:34:57.761217 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:34:57.761177 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:34:57.843271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:57.843232 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq"] Nov 02 01:34:57.881021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:57.880637 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:57.881768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:57.881661 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq"] Nov 02 01:34:57.910429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:57.910400 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-677312-serving-cert\"" Nov 02 01:34:58.010082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.010031 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nggb\" (UniqueName: \"kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.010271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.010139 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.010271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.010214 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.010401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.010381 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.215212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.137633 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.415207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.278736 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.415207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.408217 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.427111 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.427081 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.446151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.445668 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nggb\" (UniqueName: \"kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.446367 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.428110 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.490143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.476793 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.511685 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.494822 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nggb\" (UniqueName: \"kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb\") pod \"maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.512206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.511936 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:58.881937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:58.881662 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq"] Nov 02 01:34:59.710306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:59.710270 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" event={"ID":"f7ab0028-163a-43e4-950d-69c3b2f8c4f9","Type":"ContainerStarted","Data":"67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8"} Nov 02 01:34:59.710471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:59.710337 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" event={"ID":"f7ab0028-163a-43e4-950d-69c3b2f8c4f9","Type":"ContainerStarted","Data":"3c63bb97af84f85884c298debe244749500a83899c2e0e8a44e2e3cae00cf26b"} Nov 02 01:34:59.710602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:59.710580 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:34:59.733783 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:34:59.733721 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" podStartSLOduration=2.097948349 podStartE2EDuration="2.733696287s" totalImagesPullingTime="635.747938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:59.729779762 +0000 UTC m=+1199.835568677" watchObservedRunningTime="2026-11-02 01:34:59.733696287 +0000 UTC m=+1199.839485202" Nov 02 01:35:01.740595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:01.740555 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6589-d2cedc" event={"ID":"52cc646b-5758-438f-a0ac-97d20025b54c","Type":"ContainerStarted","Data":"93f7be736d929e1bb4d868a8ec252980f93c0d6238894f7ad30c105f63e833f7"} Nov 02 01:35:01.757988 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:01.757936 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6589-d2cedc" podStartSLOduration=2.200304381 podStartE2EDuration="7.757920274s" totalImagesPullingTime="5.557615893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:01.756511813 +0000 UTC m=+1201.862300727" watchObservedRunningTime="2026-11-02 01:35:01.757920274 +0000 UTC m=+1201.863709187" Nov 02 01:35:02.477678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:02.477611 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6589-d2cedc"] Nov 02 01:35:02.484895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:02.484846 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6589-d2cedc"] Nov 02 01:35:03.759718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:03.759665 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6589-d2cedc" podUID="52cc646b-5758-438f-a0ac-97d20025b54c" containerName="test-curl-6589-d2cedc" containerID="cri-o://93f7be736d929e1bb4d868a8ec252980f93c0d6238894f7ad30c105f63e833f7" gracePeriod=2 Nov 02 01:35:03.762430 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:03.762385 4552 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6589-d2cedc\" is forbidden: User \"system:node:ip-10-0-137-1.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-east-2.compute.internal' and this object" podUID="52cc646b-5758-438f-a0ac-97d20025b54c" pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:35:05.728850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:05.728818 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:35:05.807967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:05.807928 4552 generic.go:356] "Generic (PLEG): container finished" podID="52cc646b-5758-438f-a0ac-97d20025b54c" containerID="93f7be736d929e1bb4d868a8ec252980f93c0d6238894f7ad30c105f63e833f7" exitCode=137 Nov 02 01:35:06.002982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:06.002906 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:35:06.090153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:06.089419 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs\" (UniqueName: \"kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs\") pod \"52cc646b-5758-438f-a0ac-97d20025b54c\" (UID: \"52cc646b-5758-438f-a0ac-97d20025b54c\") " Nov 02 01:35:06.120580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:06.115962 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs" pod "52cc646b-5758-438f-a0ac-97d20025b54c" (UID: "52cc646b-5758-438f-a0ac-97d20025b54c"). InnerVolumeSpecName "kube-api-access-zwzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:06.192410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:06.192371 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzrs\" (UniqueName: \"kubernetes.io/projected/52cc646b-5758-438f-a0ac-97d20025b54c-kube-api-access-zwzrs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:06.818072 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:06.818044 4552 scope.go:122] "RemoveContainer" containerID="93f7be736d929e1bb4d868a8ec252980f93c0d6238894f7ad30c105f63e833f7" Nov 02 01:35:06.818560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:06.818204 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-d2cedc" Nov 02 01:35:07.019964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.019930 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx"] Nov 02 01:35:07.020821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.020800 4552 cpu_manager.go:414] "containerMap: removing container" podUID="52cc646b-5758-438f-a0ac-97d20025b54c" containerName="test-curl-6589-d2cedc" Nov 02 01:35:07.020821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.020820 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cc646b-5758-438f-a0ac-97d20025b54c" containerName="test-curl-6589-d2cedc" Nov 02 01:35:07.021010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.020943 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cc646b-5758-438f-a0ac-97d20025b54c" containerName="test-curl-6589-d2cedc" Nov 02 01:35:07.028491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.028445 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.038991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.038960 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-bbb385-serving-cert\"" Nov 02 01:35:07.043657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.043501 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx"] Nov 02 01:35:07.120604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.120504 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spbn\" (UniqueName: \"kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.120789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.120608 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.120789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.120661 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.120990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.120968 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.237972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.231012 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.237972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.231357 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spbn\" (UniqueName: \"kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.237972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.231428 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.237972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.231484 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.259214 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.255728 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.281143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.272337 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.309549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.307456 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spbn\" (UniqueName: \"kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.321172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.319028 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.362977 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.362679 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:07.776470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.776429 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx"] Nov 02 01:35:07.849664 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:07.846421 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" event={"ID":"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26","Type":"ContainerStarted","Data":"f49cb9bb301ddde19d75a424aa754bef5329a20889ae890bc468e5589cc9ce86"} Nov 02 01:35:08.046584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:08.046501 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52cc646b-5758-438f-a0ac-97d20025b54c" path="/var/lib/kubelet/pods/52cc646b-5758-438f-a0ac-97d20025b54c/volumes" Nov 02 01:35:08.858248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:08.858209 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" event={"ID":"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26","Type":"ContainerStarted","Data":"0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1"} Nov 02 01:35:08.858672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:08.858283 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:08.882795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:08.882738 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" podStartSLOduration=2.3502798990000002 podStartE2EDuration="2.882724024s" totalImagesPullingTime="532.444125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:08.87800975 +0000 UTC m=+1208.983798666" watchObservedRunningTime="2026-11-02 01:35:08.882724024 +0000 UTC m=+1208.988512938" Nov 02 01:35:11.044958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.044925 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6"] Nov 02 01:35:11.046040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.046012 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" podUID="664684ad-781b-4763-9b67-d1506dac2664" containerName="maas-api" containerID="cri-o://8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9" gracePeriod=30 Nov 02 01:35:11.468143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.460280 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:35:11.668437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.666799 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls\") pod \"664684ad-781b-4763-9b67-d1506dac2664\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " Nov 02 01:35:11.668437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.667043 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw\" (UniqueName: \"kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw\") pod \"664684ad-781b-4763-9b67-d1506dac2664\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " Nov 02 01:35:11.668437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.667141 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs\") pod \"664684ad-781b-4763-9b67-d1506dac2664\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " Nov 02 01:35:11.668437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.667274 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca\") pod \"664684ad-781b-4763-9b67-d1506dac2664\" (UID: \"664684ad-781b-4763-9b67-d1506dac2664\") " Nov 02 01:35:11.668812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.668677 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca" pod "664684ad-781b-4763-9b67-d1506dac2664" (UID: "664684ad-781b-4763-9b67-d1506dac2664"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:11.700389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.700320 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw" pod "664684ad-781b-4763-9b67-d1506dac2664" (UID: "664684ad-781b-4763-9b67-d1506dac2664"). InnerVolumeSpecName "kube-api-access-l5dgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:11.704142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.702670 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls" pod "664684ad-781b-4763-9b67-d1506dac2664" (UID: "664684ad-781b-4763-9b67-d1506dac2664"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:35:11.704142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.703215 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs" pod "664684ad-781b-4763-9b67-d1506dac2664" (UID: "664684ad-781b-4763-9b67-d1506dac2664"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:35:11.771106 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.771066 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:11.771106 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.771101 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5dgw\" (UniqueName: \"kubernetes.io/projected/664684ad-781b-4763-9b67-d1506dac2664-kube-api-access-l5dgw\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:11.771106 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.771132 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664684ad-781b-4763-9b67-d1506dac2664-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:11.771106 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.771145 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/664684ad-781b-4763-9b67-d1506dac2664-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:11.887050 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.887011 4552 generic.go:356] "Generic (PLEG): container finished" podID="664684ad-781b-4763-9b67-d1506dac2664" containerID="8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9" exitCode=0 Nov 02 01:35:11.887238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.887065 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" Nov 02 01:35:11.887238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.887077 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" event={"ID":"664684ad-781b-4763-9b67-d1506dac2664","Type":"ContainerDied","Data":"8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9"} Nov 02 01:35:11.887238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.887150 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6" event={"ID":"664684ad-781b-4763-9b67-d1506dac2664","Type":"ContainerDied","Data":"15897306ace0bc98ef4e28d778d5982452b312b78794f4d7a9b8f970a08cc2ad"} Nov 02 01:35:11.887238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.887181 4552 scope.go:122] "RemoveContainer" containerID="8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9" Nov 02 01:35:11.898984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.898961 4552 scope.go:122] "RemoveContainer" containerID="8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9" Nov 02 01:35:11.899308 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:11.899290 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9\": container with ID starting with 8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9 not found: ID does not exist" containerID="8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9" Nov 02 01:35:11.899366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.899316 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9"} err="failed to get container status \"8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9\": rpc error: code = NotFound desc = could not find container \"8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9\": container with ID starting with 8f77afdfedb84d31c56fe936ed886da3cf2866e448e72e054270797afa26dde9 not found: ID does not exist" Nov 02 01:35:11.912770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.912745 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6"] Nov 02 01:35:11.917267 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:11.917248 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b78d-584f65b4d7-hp9b6"] Nov 02 01:35:12.079139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:12.078105 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664684ad-781b-4763-9b67-d1506dac2664" path="/var/lib/kubelet/pods/664684ad-781b-4763-9b67-d1506dac2664/volumes" Nov 02 01:35:14.873665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:14.873635 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:35:15.845802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:15.845763 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:35:16.384719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.384687 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb"] Nov 02 01:35:16.386070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.386038 4552 cpu_manager.go:414] "containerMap: removing container" podUID="664684ad-781b-4763-9b67-d1506dac2664" containerName="maas-api" Nov 02 01:35:16.386070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.386068 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664684ad-781b-4763-9b67-d1506dac2664" containerName="maas-api" Nov 02 01:35:16.386337 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.386321 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="664684ad-781b-4763-9b67-d1506dac2664" containerName="maas-api" Nov 02 01:35:16.394685 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.394597 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.397686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.397624 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb"] Nov 02 01:35:16.405086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.403971 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0bbeda-serving-cert\"" Nov 02 01:35:16.495053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.495003 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.495285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.495179 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.495422 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.495398 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fg57\" (UniqueName: \"kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.502589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.502259 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.617596 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.611291 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.623519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.623469 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.623793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.623770 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fg57\" (UniqueName: \"kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.623886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.623819 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.624079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.624056 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.637060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.636982 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.638341 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.638279 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.662465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.661851 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fg57\" (UniqueName: \"kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57\") pod \"maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:16.811109 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:16.811074 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:17.120670 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:35:17.120637 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7dce9b_fb14_4c23_8960_11ea4bcd109a.slice/crio-85c03d3ddf1087f5bdcd4d674edcd1070b84fe8135fdba75536324340ef4c80a WatchSource:0}: Error finding container 85c03d3ddf1087f5bdcd4d674edcd1070b84fe8135fdba75536324340ef4c80a: Status 404 returned error can't find the container with id 85c03d3ddf1087f5bdcd4d674edcd1070b84fe8135fdba75536324340ef4c80a Nov 02 01:35:17.121222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:17.121201 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb"] Nov 02 01:35:18.040144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:18.027518 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" event={"ID":"fd7dce9b-fb14-4c23-8960-11ea4bcd109a","Type":"ContainerStarted","Data":"85c03d3ddf1087f5bdcd4d674edcd1070b84fe8135fdba75536324340ef4c80a"} Nov 02 01:35:18.865144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:18.856274 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:18.997804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:18.982222 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:19.111780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.110430 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" event={"ID":"fd7dce9b-fb14-4c23-8960-11ea4bcd109a","Type":"ContainerStarted","Data":"0d3c4d4a4025cdbbd36241b2ff231786d472eb893f0031e02d8f5767a70be19a"} Nov 02 01:35:19.111780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.111704 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:19.250192 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.249300 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:35:19.250192 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.249370 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:19.700206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.697485 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:19.829154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.820258 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:19.857608 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.857561 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:19.858777 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.858753 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:35:19.905142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:19.899414 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:35:20.222418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:20.220894 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" podStartSLOduration=3.673975791 podStartE2EDuration="4.220871801s" totalImagesPullingTime="546.89601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:19.165303396 +0000 UTC m=+1219.271092312" watchObservedRunningTime="2026-11-02 01:35:20.220871801 +0000 UTC m=+1220.326660715" Nov 02 01:35:20.344145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:20.343164 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:35:26.400596 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:26.400564 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:35:27.818142 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:27.799963 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:35:30.997066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:30.997033 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm"] Nov 02 01:35:31.006353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.005478 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.014389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.014363 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm"] Nov 02 01:35:31.074423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.074397 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0e06f2-serving-cert\"" Nov 02 01:35:31.121841 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.121810 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.122075 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.122052 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2rf\" (UniqueName: \"kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.122160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.122147 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.122207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.122184 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.233698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.233648 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.234000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.233972 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2rf\" (UniqueName: \"kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.234165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.234072 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.234165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.234101 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.238086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.238059 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.262200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.250911 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.277233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.267997 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.277233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.276500 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2rf\" (UniqueName: \"kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf\") pod \"maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.332141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.327954 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:31.758058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.755528 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm"] Nov 02 01:35:31.758485 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:35:31.758440 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd03964c_70f0_4f6d_b23f_3e4e09650784.slice/crio-7dba85fc27e57d8d89f494fe83507ccc20f7affebc13e2141025ff1c22391781 WatchSource:0}: Error finding container 7dba85fc27e57d8d89f494fe83507ccc20f7affebc13e2141025ff1c22391781: Status 404 returned error can't find the container with id 7dba85fc27e57d8d89f494fe83507ccc20f7affebc13e2141025ff1c22391781 Nov 02 01:35:31.769004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:31.768982 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:35:32.786549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:32.786470 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" event={"ID":"fd03964c-70f0-4f6d-b23f-3e4e09650784","Type":"ContainerStarted","Data":"11b9b31de8b816531befd07ae2884c2baef00e9614763ebc73d955d1fe336a4e"} Nov 02 01:35:32.786549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:32.786546 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" event={"ID":"fd03964c-70f0-4f6d-b23f-3e4e09650784","Type":"ContainerStarted","Data":"7dba85fc27e57d8d89f494fe83507ccc20f7affebc13e2141025ff1c22391781"} Nov 02 01:35:32.787003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:32.786630 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:32.809187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:32.809081 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" podStartSLOduration=2.2000354619999998 podStartE2EDuration="2.809067115s" totalImagesPullingTime="609.031653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:32.806448371 +0000 UTC m=+1232.912237283" watchObservedRunningTime="2026-11-02 01:35:32.809067115 +0000 UTC m=+1232.914856061" Nov 02 01:35:38.803841 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:38.803806 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:35:40.264612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.264581 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b"] Nov 02 01:35:40.273238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.273216 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.279401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.279374 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-addad9-serving-cert\"" Nov 02 01:35:40.285494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.285468 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b"] Nov 02 01:35:40.333559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.333527 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.333759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.333587 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.333759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.333702 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.333759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.333739 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4z7\" (UniqueName: \"kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.434724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.434693 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.434917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.434747 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.434917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.434848 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.434917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.434875 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4z7\" (UniqueName: \"kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.435619 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.435546 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.437576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.437550 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.437698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.437553 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.447532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.447505 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4z7\" (UniqueName: \"kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7\") pod \"maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.608235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.608091 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:40.959994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:40.953350 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b"] Nov 02 01:35:41.956982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:41.956857 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" event={"ID":"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8","Type":"ContainerStarted","Data":"adf10848209418d4380ee774f22066c71e25baeae365b1e6f8ed55bac8ab0b91"} Nov 02 01:35:41.957698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:41.957670 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" event={"ID":"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8","Type":"ContainerStarted","Data":"40eed1681327aa099bd4f704f064e5deae28fb41e08de8936d1852b846075c40"} Nov 02 01:35:41.959395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:41.959372 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:41.985851 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:41.985798 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" podStartSLOduration=1.267465839 podStartE2EDuration="1.985779511s" totalImagesPullingTime="718.313672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:41.979693128 +0000 UTC m=+1242.085482044" watchObservedRunningTime="2026-11-02 01:35:41.985779511 +0000 UTC m=+1242.091568426" Nov 02 01:35:42.520555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:42.520519 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq"] Nov 02 01:35:42.521132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:42.520795 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" podUID="704acddf-addc-4042-bdd4-a69edcc710ff" containerName="maas-api" containerID="cri-o://c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0" gracePeriod=30 Nov 02 01:35:42.955355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:42.955325 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:35:43.094190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.084087 4552 generic.go:356] "Generic (PLEG): container finished" podID="704acddf-addc-4042-bdd4-a69edcc710ff" containerID="c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0" exitCode=0 Nov 02 01:35:43.094190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.085480 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" Nov 02 01:35:43.094190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.085958 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" event={"ID":"704acddf-addc-4042-bdd4-a69edcc710ff","Type":"ContainerDied","Data":"c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0"} Nov 02 01:35:43.094190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.086031 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq" event={"ID":"704acddf-addc-4042-bdd4-a69edcc710ff","Type":"ContainerDied","Data":"29f49faa4119dd574fb5b3c9c1b4d3f7b6d37f56b9201c8774506ba46f67c1cb"} Nov 02 01:35:43.094190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.086067 4552 scope.go:122] "RemoveContainer" containerID="c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0" Nov 02 01:35:43.137333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.137214 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs\") pod \"704acddf-addc-4042-bdd4-a69edcc710ff\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " Nov 02 01:35:43.138039 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.138008 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca" pod "704acddf-addc-4042-bdd4-a69edcc710ff" (UID: "704acddf-addc-4042-bdd4-a69edcc710ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:43.141219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.140549 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca\") pod \"704acddf-addc-4042-bdd4-a69edcc710ff\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " Nov 02 01:35:43.141219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.140717 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c\" (UniqueName: \"kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c\") pod \"704acddf-addc-4042-bdd4-a69edcc710ff\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " Nov 02 01:35:43.141746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.141641 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls\") pod \"704acddf-addc-4042-bdd4-a69edcc710ff\" (UID: \"704acddf-addc-4042-bdd4-a69edcc710ff\") " Nov 02 01:35:43.150535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.150500 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/704acddf-addc-4042-bdd4-a69edcc710ff-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:43.153161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.153092 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls" pod "704acddf-addc-4042-bdd4-a69edcc710ff" (UID: "704acddf-addc-4042-bdd4-a69edcc710ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:35:43.153636 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.153581 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs" pod "704acddf-addc-4042-bdd4-a69edcc710ff" (UID: "704acddf-addc-4042-bdd4-a69edcc710ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:35:43.155379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.155356 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c" pod "704acddf-addc-4042-bdd4-a69edcc710ff" (UID: "704acddf-addc-4042-bdd4-a69edcc710ff"). InnerVolumeSpecName "kube-api-access-6kb5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:43.247400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.244876 4552 scope.go:122] "RemoveContainer" containerID="c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0" Nov 02 01:35:43.247400 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:43.245420 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0\": container with ID starting with c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0 not found: ID does not exist" containerID="c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0" Nov 02 01:35:43.247400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.245459 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0"} err="failed to get container status \"c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0\": rpc error: code = NotFound desc = could not find container \"c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0\": container with ID starting with c551255b6bd81aca8569418648615406d07b166a4fcfc47d85a782b6702521b0 not found: ID does not exist" Nov 02 01:35:43.254161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.252686 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:43.254161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.252704 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/704acddf-addc-4042-bdd4-a69edcc710ff-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:43.254161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.252715 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kb5c\" (UniqueName: \"kubernetes.io/projected/704acddf-addc-4042-bdd4-a69edcc710ff-kube-api-access-6kb5c\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:43.417154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.417079 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq"] Nov 02 01:35:43.422189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:43.422167 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-2rnkq"] Nov 02 01:35:44.060988 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:44.060945 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704acddf-addc-4042-bdd4-a69edcc710ff" path="/var/lib/kubelet/pods/704acddf-addc-4042-bdd4-a69edcc710ff/volumes" Nov 02 01:35:47.190588 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.190545 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k"] Nov 02 01:35:47.193451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.192733 4552 cpu_manager.go:414] "containerMap: removing container" podUID="704acddf-addc-4042-bdd4-a69edcc710ff" containerName="maas-api" Nov 02 01:35:47.193451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.192758 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704acddf-addc-4042-bdd4-a69edcc710ff" containerName="maas-api" Nov 02 01:35:47.193451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.193084 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="704acddf-addc-4042-bdd4-a69edcc710ff" containerName="maas-api" Nov 02 01:35:47.211104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.211072 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k"] Nov 02 01:35:47.211482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.211447 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.219667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.219045 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-00f1da-serving-cert\"" Nov 02 01:35:47.266652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.266611 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.273801 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.273766 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zl9\" (UniqueName: \"kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.274099 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.274073 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.274432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.274411 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.387590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.383519 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.387590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.383690 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zl9\" (UniqueName: \"kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.387590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.383745 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.387590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.383830 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.388511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.388477 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.390199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.390175 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.391617 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.391601 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.400904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.400886 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zl9\" (UniqueName: \"kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9\") pod \"maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.533487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.533445 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:47.950227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:47.947739 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k"] Nov 02 01:35:48.143388 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:48.143341 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" event={"ID":"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033","Type":"ContainerStarted","Data":"d85348c9329561b5835a30dce98cbac37c5940384082a6f33e1c1e9e3e593b3d"} Nov 02 01:35:49.101487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:49.101459 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:35:49.159860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:49.159819 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" event={"ID":"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033","Type":"ContainerStarted","Data":"21ba71f4ab39f4b641c9cf63d789a36dc4bef6d78caa059a9e1dfe7dc0a6a236"} Nov 02 01:35:49.161086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:49.161059 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:49.193490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:49.193433 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" podStartSLOduration=1.682736633 podStartE2EDuration="2.193400588s" totalImagesPullingTime="510.663955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:49.180714776 +0000 UTC m=+1249.286503688" watchObservedRunningTime="2026-11-02 01:35:49.193400588 +0000 UTC m=+1249.299189502" Nov 02 01:35:55.260154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.237924 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc"] Nov 02 01:35:55.260154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.247555 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.260154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.253619 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ffe5c1-serving-cert\"" Nov 02 01:35:55.275250 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.274957 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc"] Nov 02 01:35:55.416144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.409532 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.416144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.409598 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.416144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.409728 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6f9w\" (UniqueName: \"kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.416144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.409801 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.511930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.511820 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.511930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.511888 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.512301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.511959 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6f9w\" (UniqueName: \"kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.512301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.512017 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.512710 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:55.512688 4552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-serving-cert: secret "maas-api-e2e-inf-b-ffe5c1-serving-cert" not found Nov 02 01:35:55.512963 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:55.512941 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls podName:bfa01645-9018-4903-b68a-5afd9ca285ac nodeName:}" failed. No retries permitted until 2026-11-02 01:35:56.012916853 +0000 UTC m=+1256.118705769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls") pod "maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" (UID: "bfa01645-9018-4903-b68a-5afd9ca285ac") : secret "maas-api-e2e-inf-b-ffe5c1-serving-cert" not found Nov 02 01:35:55.513182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.512773 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.517401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.517378 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:55.525495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:55.525469 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6f9w\" (UniqueName: \"kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:56.019796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:56.019283 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:56.078980 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:56.078938 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:56.189278 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:56.189246 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:56.218972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:56.218647 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:35:56.525322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:56.525279 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc"] Nov 02 01:35:57.064077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:57.064038 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:35:57.372212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:57.371388 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" event={"ID":"bfa01645-9018-4903-b68a-5afd9ca285ac","Type":"ContainerStarted","Data":"c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b"} Nov 02 01:35:57.372212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:57.371454 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" event={"ID":"bfa01645-9018-4903-b68a-5afd9ca285ac","Type":"ContainerStarted","Data":"06f8728a41a5b61ec9385a79ca6834d249080949cd499ba5087784b27944f3c5"} Nov 02 01:35:57.382144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:57.378223 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:35:57.450142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:35:57.449579 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" podStartSLOduration=1.961888888 podStartE2EDuration="2.449557324s" totalImagesPullingTime="487.668436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:57.44143327 +0000 UTC m=+1257.547222184" watchObservedRunningTime="2026-11-02 01:35:57.449557324 +0000 UTC m=+1257.555346238" Nov 02 01:35:57.762088 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:35:57.762050 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:36:00.356248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.356215 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n"] Nov 02 01:36:00.362358 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.362333 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.367523 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.367499 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50537f4f-serving-cert\"" Nov 02 01:36:00.373498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.373474 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n"] Nov 02 01:36:00.409066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.409031 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.409275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.409162 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbcc\" (UniqueName: \"kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.409457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.409438 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.409560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.409547 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.511251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.511222 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.511430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.511305 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.511430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.511355 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.511430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.511395 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbcc\" (UniqueName: \"kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.511976 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.511946 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.514020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.514000 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.514109 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.514043 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.525968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.525943 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbcc\" (UniqueName: \"kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc\") pod \"maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:00.685473 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:00.685386 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:01.030633 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:36:01.030591 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb630033d_3b64_4ddc_9e83_e69562939b63.slice/crio-9dee2938895611da8cdc8a4facf6c96f2ad08d112beb24c2dbef5fa1eef1ac01 WatchSource:0}: Error finding container 9dee2938895611da8cdc8a4facf6c96f2ad08d112beb24c2dbef5fa1eef1ac01: Status 404 returned error can't find the container with id 9dee2938895611da8cdc8a4facf6c96f2ad08d112beb24c2dbef5fa1eef1ac01 Nov 02 01:36:01.031587 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:01.031536 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n"] Nov 02 01:36:01.424507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:01.424421 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" event={"ID":"b630033d-3b64-4ddc-9e83-e69562939b63","Type":"ContainerStarted","Data":"9dee2938895611da8cdc8a4facf6c96f2ad08d112beb24c2dbef5fa1eef1ac01"} Nov 02 01:36:02.438919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:02.438881 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" event={"ID":"b630033d-3b64-4ddc-9e83-e69562939b63","Type":"ContainerStarted","Data":"c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20"} Nov 02 01:36:02.439360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:02.438971 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:02.467451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:02.467392 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" podStartSLOduration=1.906572749 podStartE2EDuration="2.467374011s" totalImagesPullingTime="560.801262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:02.459272837 +0000 UTC m=+1262.565061781" watchObservedRunningTime="2026-11-02 01:36:02.467374011 +0000 UTC m=+1262.573162926" Nov 02 01:36:03.415138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:03.411554 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:36:08.481461 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:08.481433 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:36:14.246978 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.246944 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6586-a01be6"] Nov 02 01:36:14.253619 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.253597 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:14.261068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.261033 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-a01be6"] Nov 02 01:36:14.267488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.267165 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vhtnz\"" Nov 02 01:36:14.377705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.377666 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzkk\" (UniqueName: \"kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk\") pod \"test-curl-6586-a01be6\" (UID: \"f7d2cede-3317-40b2-a9ee-00a315b068b1\") " pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:14.479216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.479177 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzkk\" (UniqueName: \"kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk\") pod \"test-curl-6586-a01be6\" (UID: \"f7d2cede-3317-40b2-a9ee-00a315b068b1\") " pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:14.491325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.491301 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzkk\" (UniqueName: \"kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk\") pod \"test-curl-6586-a01be6\" (UID: \"f7d2cede-3317-40b2-a9ee-00a315b068b1\") " pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:14.600306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.600109 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:14.851728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.851655 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2"] Nov 02 01:36:14.855656 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.855638 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:14.885865 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.885825 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b34fc0-serving-cert\"" Nov 02 01:36:14.902852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.902827 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2"] Nov 02 01:36:14.952667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:14.952639 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-a01be6"] Nov 02 01:36:15.010140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.000899 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.010140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.001098 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.010140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.001398 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8l8n\" (UniqueName: \"kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.010140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.001642 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.103753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.103673 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.103884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.103864 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.104042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.104025 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.104093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.104066 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8l8n\" (UniqueName: \"kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.104473 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.104447 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.106539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.106516 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.106750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.106731 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.118607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.118575 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8l8n\" (UniqueName: \"kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n\") pod \"maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.175011 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.174972 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:15.488713 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:36:15.488661 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8c89bf_d8ca_45c5_8d6b_8ddac25fc0da.slice/crio-504c7f868132dabdb5ae39efd1ca2e0a852673b2c8b3d1e0c2618525377fea2a WatchSource:0}: Error finding container 504c7f868132dabdb5ae39efd1ca2e0a852673b2c8b3d1e0c2618525377fea2a: Status 404 returned error can't find the container with id 504c7f868132dabdb5ae39efd1ca2e0a852673b2c8b3d1e0c2618525377fea2a Nov 02 01:36:15.491342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.491299 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2"] Nov 02 01:36:15.591079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.591022 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-a01be6" event={"ID":"f7d2cede-3317-40b2-a9ee-00a315b068b1","Type":"ContainerStarted","Data":"36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3"} Nov 02 01:36:15.591911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.591879 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-a01be6" event={"ID":"f7d2cede-3317-40b2-a9ee-00a315b068b1","Type":"ContainerStarted","Data":"a352c8249e6b408c9a2ed2a50cc14ca5d783dbe4aede29c5ef5dd6d0380e6c42"} Nov 02 01:36:15.593527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:15.593503 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" event={"ID":"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da","Type":"ContainerStarted","Data":"504c7f868132dabdb5ae39efd1ca2e0a852673b2c8b3d1e0c2618525377fea2a"} Nov 02 01:36:16.418532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:16.418412 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6586-a01be6" podStartSLOduration=2.4183926270000002 podStartE2EDuration="2.418392627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:15.643987852 +0000 UTC m=+1275.749776745" watchObservedRunningTime="2026-11-02 01:36:16.418392627 +0000 UTC m=+1276.524181545" Nov 02 01:36:16.424083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:16.424034 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6586-a01be6"] Nov 02 01:36:16.429433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:16.429405 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6586-a01be6"] Nov 02 01:36:16.668149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:16.667093 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" event={"ID":"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da","Type":"ContainerStarted","Data":"3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8"} Nov 02 01:36:16.668149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:16.667198 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:16.700195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:16.696941 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" podStartSLOduration=2.126562475 podStartE2EDuration="2.696923386s" totalImagesPullingTime="570.360911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:16.695847525 +0000 UTC m=+1276.801636438" watchObservedRunningTime="2026-11-02 01:36:16.696923386 +0000 UTC m=+1276.802712301" Nov 02 01:36:17.673843 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:17.673799 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6586-a01be6" podUID="f7d2cede-3317-40b2-a9ee-00a315b068b1" containerName="test-curl-6586-a01be6" containerID="cri-o://36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3" gracePeriod=2 Nov 02 01:36:17.676349 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:17.676324 4552 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-a01be6\" is forbidden: User \"system:node:ip-10-0-137-1.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-east-2.compute.internal' and this object" podUID="f7d2cede-3317-40b2-a9ee-00a315b068b1" pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:19.916352 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:19.916324 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:20.025044 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.025006 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk\" (UniqueName: \"kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk\") pod \"f7d2cede-3317-40b2-a9ee-00a315b068b1\" (UID: \"f7d2cede-3317-40b2-a9ee-00a315b068b1\") " Nov 02 01:36:20.027547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.027515 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk" pod "f7d2cede-3317-40b2-a9ee-00a315b068b1" (UID: "f7d2cede-3317-40b2-a9ee-00a315b068b1"). InnerVolumeSpecName "kube-api-access-pqzkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:20.048390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.048363 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d2cede-3317-40b2-a9ee-00a315b068b1" path="/var/lib/kubelet/pods/f7d2cede-3317-40b2-a9ee-00a315b068b1/volumes" Nov 02 01:36:20.146147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.144760 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqzkk\" (UniqueName: \"kubernetes.io/projected/f7d2cede-3317-40b2-a9ee-00a315b068b1-kube-api-access-pqzkk\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:20.715616 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.715582 4552 generic.go:356] "Generic (PLEG): container finished" podID="f7d2cede-3317-40b2-a9ee-00a315b068b1" containerID="36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3" exitCode=137 Nov 02 01:36:20.715836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.715632 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-a01be6" Nov 02 01:36:20.715836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.715692 4552 scope.go:122] "RemoveContainer" containerID="36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3" Nov 02 01:36:20.729657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.729637 4552 scope.go:122] "RemoveContainer" containerID="36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3" Nov 02 01:36:20.729962 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:20.729943 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3\": container with ID starting with 36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3 not found: ID does not exist" containerID="36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3" Nov 02 01:36:20.730021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:20.729971 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3"} err="failed to get container status \"36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3\": rpc error: code = NotFound desc = could not find container \"36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3\": container with ID starting with 36daac3cede55594b6cc1e5331db547791260ea0ff122a292d57eae6ed35cfd3 not found: ID does not exist" Nov 02 01:36:22.694884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:22.694856 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:36:27.765575 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:27.765513 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:36:28.893934 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.893893 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6589-e1a3d4"] Nov 02 01:36:28.894829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.894804 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f7d2cede-3317-40b2-a9ee-00a315b068b1" containerName="test-curl-6586-a01be6" Nov 02 01:36:28.894829 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.894828 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d2cede-3317-40b2-a9ee-00a315b068b1" containerName="test-curl-6586-a01be6" Nov 02 01:36:28.895064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.895049 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d2cede-3317-40b2-a9ee-00a315b068b1" containerName="test-curl-6586-a01be6" Nov 02 01:36:28.900890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.900867 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:28.904731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.904702 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vhtnz\"" Nov 02 01:36:28.906754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.906732 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6589-e1a3d4"] Nov 02 01:36:28.967974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:28.967934 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdct\" (UniqueName: \"kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct\") pod \"test-curl-6589-e1a3d4\" (UID: \"d3fa9000-bab0-4898-b73b-ad41f8e87837\") " pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:29.069662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:29.069616 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdct\" (UniqueName: \"kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct\") pod \"test-curl-6589-e1a3d4\" (UID: \"d3fa9000-bab0-4898-b73b-ad41f8e87837\") " pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:29.082097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:29.082060 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdct\" (UniqueName: \"kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct\") pod \"test-curl-6589-e1a3d4\" (UID: \"d3fa9000-bab0-4898-b73b-ad41f8e87837\") " pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:29.234349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:29.234321 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:29.554251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:29.554220 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6589-e1a3d4"] Nov 02 01:36:29.564140 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:36:29.556157 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3fa9000_bab0_4898_b73b_ad41f8e87837.slice/crio-3fde875019c72e7ce4727540ddd7c5ecc29e2508b10f7b83b8cb63805ad97a3b WatchSource:0}: Error finding container 3fde875019c72e7ce4727540ddd7c5ecc29e2508b10f7b83b8cb63805ad97a3b: Status 404 returned error can't find the container with id 3fde875019c72e7ce4727540ddd7c5ecc29e2508b10f7b83b8cb63805ad97a3b Nov 02 01:36:29.939916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:29.939777 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6589-e1a3d4" event={"ID":"d3fa9000-bab0-4898-b73b-ad41f8e87837","Type":"ContainerStarted","Data":"7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193"} Nov 02 01:36:29.940789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:29.940757 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6589-e1a3d4" event={"ID":"d3fa9000-bab0-4898-b73b-ad41f8e87837","Type":"ContainerStarted","Data":"3fde875019c72e7ce4727540ddd7c5ecc29e2508b10f7b83b8cb63805ad97a3b"} Nov 02 01:36:30.773159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:30.768749 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6589-e1a3d4" podStartSLOduration=2.768730426 podStartE2EDuration="2.768730426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:29.95835169 +0000 UTC m=+1290.064140620" watchObservedRunningTime="2026-11-02 01:36:30.768730426 +0000 UTC m=+1290.874519339" Nov 02 01:36:30.797458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:30.796451 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6589-e1a3d4"] Nov 02 01:36:30.797458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:30.796524 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6589-e1a3d4"] Nov 02 01:36:31.963289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:31.963243 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6589-e1a3d4" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" containerName="test-curl-6589-e1a3d4" containerID="cri-o://7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193" gracePeriod=2 Nov 02 01:36:31.966244 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:31.966206 4552 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6589-e1a3d4\" is forbidden: User \"system:node:ip-10-0-137-1.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-east-2.compute.internal' and this object" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:34.263769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:34.263734 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:34.312686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:34.312656 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct\" (UniqueName: \"kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct\") pod \"d3fa9000-bab0-4898-b73b-ad41f8e87837\" (UID: \"d3fa9000-bab0-4898-b73b-ad41f8e87837\") " Nov 02 01:36:34.315001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:34.314975 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct" pod "d3fa9000-bab0-4898-b73b-ad41f8e87837" (UID: "d3fa9000-bab0-4898-b73b-ad41f8e87837"). InnerVolumeSpecName "kube-api-access-vfdct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:34.414382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:34.414342 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfdct\" (UniqueName: \"kubernetes.io/projected/d3fa9000-bab0-4898-b73b-ad41f8e87837-kube-api-access-vfdct\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:35.002552 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:35.002513 4552 generic.go:356] "Generic (PLEG): container finished" podID="d3fa9000-bab0-4898-b73b-ad41f8e87837" containerID="7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193" exitCode=137 Nov 02 01:36:35.002706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:35.002579 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:35.002706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:35.002623 4552 scope.go:122] "RemoveContainer" containerID="7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193" Nov 02 01:36:35.020137 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:35.018277 4552 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6589-e1a3d4\" is forbidden: User \"system:node:ip-10-0-137-1.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-east-2.compute.internal' and this object" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" pod="opendatahub/test-curl-6589-e1a3d4" Nov 02 01:36:35.020137 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:35.020041 4552 scope.go:122] "RemoveContainer" containerID="7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193" Nov 02 01:36:35.022494 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:35.020553 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193\": container with ID starting with 7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193 not found: ID does not exist" containerID="7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193" Nov 02 01:36:35.022494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:35.020597 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193"} err="failed to get container status \"7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193\": rpc error: code = NotFound desc = could not find container \"7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193\": container with ID starting with 7f3a4ea7184f2c9d9eb3c8bee082e3322a3eeb01c28120947d108f5fda7dd193 not found: ID does not exist" Nov 02 01:36:36.049075 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:36.049044 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" path="/var/lib/kubelet/pods/d3fa9000-bab0-4898-b73b-ad41f8e87837/volumes" Nov 02 01:36:38.671481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.671444 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6"] Nov 02 01:36:38.673229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.673203 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" containerName="test-curl-6589-e1a3d4" Nov 02 01:36:38.673229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.673228 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" containerName="test-curl-6589-e1a3d4" Nov 02 01:36:38.673434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.673419 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fa9000-bab0-4898-b73b-ad41f8e87837" containerName="test-curl-6589-e1a3d4" Nov 02 01:36:38.679027 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.679005 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.690924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.690882 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffe5c1\"/\"default-dockercfg-j549h\"" Nov 02 01:36:38.691951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.691227 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffe5c1\"/\"openshift-service-ca.crt\"" Nov 02 01:36:38.692149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.691806 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffe5c1\"/\"kube-root-ca.crt\"" Nov 02 01:36:38.692272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.691879 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffe5c1\"/\"test-model-ffe5c1-kserve-self-signed-certs\"" Nov 02 01:36:38.697840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.697814 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6"] Nov 02 01:36:38.731339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.731306 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.731489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.731356 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.731489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.731443 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.731589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.731571 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.731840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.731817 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.731900 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.731884 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk58t\" (UniqueName: \"kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.832957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.832928 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.832999 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk58t\" (UniqueName: \"kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833035 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833061 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833210 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833328 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833501 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833787 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.833919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.833827 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.835975 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.835952 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.836372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.836352 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:38.848623 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:38.848598 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk58t\" (UniqueName: \"kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t\") pod \"test-model-ffe5c1-kserve-697494d8c-h5ld6\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:39.010824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:39.010781 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:39.373017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:39.367786 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6"] Nov 02 01:36:40.079942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.079898 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" event={"ID":"9a1ffeff-fbf6-4319-b0ae-2f934090af33","Type":"ContainerStarted","Data":"043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe"} Nov 02 01:36:40.079942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.079949 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" event={"ID":"9a1ffeff-fbf6-4319-b0ae-2f934090af33","Type":"ContainerStarted","Data":"71cf2bb1b0d3803cace6f35e50371c0cfa37997dd526b3a481808d14878a42d6"} Nov 02 01:36:40.100159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.100074 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" podStartSLOduration=2.100059582 podStartE2EDuration="2.100059582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:40.098671442 +0000 UTC m=+1300.204460358" watchObservedRunningTime="2026-11-02 01:36:40.100059582 +0000 UTC m=+1300.205848495" Nov 02 01:36:40.177263 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.177226 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h"] Nov 02 01:36:40.177467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.177445 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" containerID="cri-o://47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107" gracePeriod=30 Nov 02 01:36:40.285998 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.278572 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf"] Nov 02 01:36:40.287810 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.287770 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" containerID="cri-o://5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b" gracePeriod=30 Nov 02 01:36:40.289917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.289889 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf"] Nov 02 01:36:40.290160 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:40.290095 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" containerID="cri-o://3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0" gracePeriod=30 Nov 02 01:36:41.012263 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:41.012226 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:42.032189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:42.032154 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:36:45.523415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.521558 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:36:45.669813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.669780 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.672206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.671993 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672535 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672644 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672709 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672838 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672865 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672895 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.672960 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs\") pod \"773a918a-ee3a-4991-bc33-5ac626a65237\" (UID: \"773a918a-ee3a-4991-bc33-5ac626a65237\") " Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.674780 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.675455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.675231 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.676465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.675649 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:45.676465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.675866 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.676465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.675926 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.677830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.677712 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.677830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.677748 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.677830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.677781 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.677830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.677795 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773a918a-ee3a-4991-bc33-5ac626a65237-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.677830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.677809 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.678688 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.678652 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "kube-api-access-fcq9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:45.682848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.682811 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.683983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.683947 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:45.685534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.685326 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo" pod "773a918a-ee3a-4991-bc33-5ac626a65237" (UID: "773a918a-ee3a-4991-bc33-5ac626a65237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:45.707223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.706824 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:36:45.778919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.778883 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.778937 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779014 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779220 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779064 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779220 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779131 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.779220 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779180 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779259 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779378 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779436 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779467 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:45.779639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779620 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo\") pod \"10383294-673a-40a9-ab9b-6ac98c648a09\" (UID: \"10383294-673a-40a9-ab9b-6ac98c648a09\") " Nov 02 01:36:45.779697 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.779678 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780176 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780509 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780527 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773a918a-ee3a-4991-bc33-5ac626a65237-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780542 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcq9g\" (UniqueName: \"kubernetes.io/projected/773a918a-ee3a-4991-bc33-5ac626a65237-kube-api-access-fcq9g\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780559 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773a918a-ee3a-4991-bc33-5ac626a65237-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780571 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780586 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780598 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10383294-673a-40a9-ab9b-6ac98c648a09-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.780654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.780611 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.781248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.781221 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.783606 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.783581 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.783760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.783735 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:45.784830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.784804 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "kube-api-access-wcr8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:45.785009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.784987 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo" pod "10383294-673a-40a9-ab9b-6ac98c648a09" (UID: "10383294-673a-40a9-ab9b-6ac98c648a09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:45.785596 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.785569 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:36:45.881704 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.881663 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.881920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.881784 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.881920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.881867 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.881920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.881894 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.882083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882009 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.882083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882040 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.882207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882100 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.882207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882159 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.882340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882323 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data\") pod \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\" (UID: \"371fa9aa-2a3a-405e-8f0e-eb05a57b842c\") " Nov 02 01:36:45.882990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882448 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.883100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882531 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.883100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882742 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.883100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882781 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:45.883100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.882894 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.884058 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884032 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884060 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884074 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884087 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcr8s\" (UniqueName: \"kubernetes.io/projected/10383294-673a-40a9-ab9b-6ac98c648a09-kube-api-access-wcr8s\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884102 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884133 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884146 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10383294-673a-40a9-ab9b-6ac98c648a09-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884159 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10383294-673a-40a9-ab9b-6ac98c648a09-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884172 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884186 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.884840 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884817 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:45.884901 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.884858 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:45.885240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.885224 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "kube-api-access-7gqgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:45.885545 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.885524 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy" pod "371fa9aa-2a3a-405e-8f0e-eb05a57b842c" (UID: "371fa9aa-2a3a-405e-8f0e-eb05a57b842c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:45.986247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.986216 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.986247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.986243 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.986247 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.986253 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:45.986535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:45.986264 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqgv\" (UniqueName: \"kubernetes.io/projected/371fa9aa-2a3a-405e-8f0e-eb05a57b842c-kube-api-access-7gqgv\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.190187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.190137 4552 generic.go:356] "Generic (PLEG): container finished" podID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerID="5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b" exitCode=0 Nov 02 01:36:46.190360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.190258 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" event={"ID":"371fa9aa-2a3a-405e-8f0e-eb05a57b842c","Type":"ContainerDied","Data":"5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b"} Nov 02 01:36:46.190360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.190323 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" event={"ID":"371fa9aa-2a3a-405e-8f0e-eb05a57b842c","Type":"ContainerDied","Data":"4df249e26cc78fc82a34119ee4d691dd8583c7da62d002517794099f4ca9a143"} Nov 02 01:36:46.190360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.190357 4552 scope.go:122] "RemoveContainer" containerID="5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b" Nov 02 01:36:46.190542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.190273 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf" Nov 02 01:36:46.191989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.191968 4552 generic.go:356] "Generic (PLEG): container finished" podID="773a918a-ee3a-4991-bc33-5ac626a65237" containerID="47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107" exitCode=0 Nov 02 01:36:46.192248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.192223 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" event={"ID":"773a918a-ee3a-4991-bc33-5ac626a65237","Type":"ContainerDied","Data":"47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107"} Nov 02 01:36:46.192567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.192273 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" event={"ID":"773a918a-ee3a-4991-bc33-5ac626a65237","Type":"ContainerDied","Data":"2323697036d1e23dd594e6a3917be5fbaf778881b9508dd29441d8022a402e77"} Nov 02 01:36:46.192567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.192361 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h" Nov 02 01:36:46.201098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.200913 4552 generic.go:356] "Generic (PLEG): container finished" podID="10383294-673a-40a9-ab9b-6ac98c648a09" containerID="3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0" exitCode=0 Nov 02 01:36:46.201098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.201004 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" Nov 02 01:36:46.201098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.201012 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" event={"ID":"10383294-673a-40a9-ab9b-6ac98c648a09","Type":"ContainerDied","Data":"3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0"} Nov 02 01:36:46.201098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.201053 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf" event={"ID":"10383294-673a-40a9-ab9b-6ac98c648a09","Type":"ContainerDied","Data":"53b6165585927ec47491a65ddbc50b2536f7f61eb3c59b09a3d76fad4185cdf2"} Nov 02 01:36:46.216350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.216326 4552 scope.go:122] "RemoveContainer" containerID="5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b" Nov 02 01:36:46.217397 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:46.217358 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b\": container with ID starting with 5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b not found: ID does not exist" containerID="5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b" Nov 02 01:36:46.217547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.217401 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b"} err="failed to get container status \"5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b\": rpc error: code = NotFound desc = could not find container \"5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b\": container with ID starting with 5733b0ba1194023194308d9c41f31f83faa86523a6ab3bca3346be2d9efeed9b not found: ID does not exist" Nov 02 01:36:46.217547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.217423 4552 scope.go:122] "RemoveContainer" containerID="47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107" Nov 02 01:36:46.270540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.270509 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf"] Nov 02 01:36:46.274966 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.274085 4552 scope.go:122] "RemoveContainer" containerID="47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107" Nov 02 01:36:46.275548 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:46.275524 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107\": container with ID starting with 47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107 not found: ID does not exist" containerID="47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107" Nov 02 01:36:46.275758 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.275729 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107"} err="failed to get container status \"47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107\": rpc error: code = NotFound desc = could not find container \"47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107\": container with ID starting with 47a1f2d6d5d7dcc8fe195f57b3e57e99b53f80c78b02fcc3d72e1a1e20e82107 not found: ID does not exist" Nov 02 01:36:46.275859 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.275846 4552 scope.go:122] "RemoveContainer" containerID="3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0" Nov 02 01:36:46.281285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.281265 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-8bnwf"] Nov 02 01:36:46.288584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.288566 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h"] Nov 02 01:36:46.296035 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.295980 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-sgn4h"] Nov 02 01:36:46.304686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.304657 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf"] Nov 02 01:36:46.305346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.305325 4552 scope.go:122] "RemoveContainer" containerID="3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0" Nov 02 01:36:46.305965 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:46.305936 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0\": container with ID starting with 3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0 not found: ID does not exist" containerID="3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0" Nov 02 01:36:46.306071 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.305974 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0"} err="failed to get container status \"3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0\": rpc error: code = NotFound desc = could not find container \"3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0\": container with ID starting with 3e1cf39e677c2379d99a090e9826d4eb94121836a7098c936de1ca14bbc6b8f0 not found: ID does not exist" Nov 02 01:36:46.311551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:46.311493 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-z6qpf"] Nov 02 01:36:48.049480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:48.049443 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" path="/var/lib/kubelet/pods/10383294-673a-40a9-ab9b-6ac98c648a09/volumes" Nov 02 01:36:48.049985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:48.049965 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" path="/var/lib/kubelet/pods/371fa9aa-2a3a-405e-8f0e-eb05a57b842c/volumes" Nov 02 01:36:48.050438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:48.050417 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" path="/var/lib/kubelet/pods/773a918a-ee3a-4991-bc33-5ac626a65237/volumes" Nov 02 01:36:49.012263 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:49.012223 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:49.025085 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:49.025064 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:49.283885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:49.283786 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:36:54.631812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.631777 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq"] Nov 02 01:36:54.632888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.632867 4552 cpu_manager.go:414] "containerMap: removing container" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" Nov 02 01:36:54.632888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.632888 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" Nov 02 01:36:54.633048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.632906 4552 cpu_manager.go:414] "containerMap: removing container" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" Nov 02 01:36:54.633048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.632913 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" Nov 02 01:36:54.633048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.632925 4552 cpu_manager.go:414] "containerMap: removing container" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" Nov 02 01:36:54.633048 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.632933 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" Nov 02 01:36:54.633295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.633136 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="371fa9aa-2a3a-405e-8f0e-eb05a57b842c" containerName="istio-proxy" Nov 02 01:36:54.633295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.633148 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="10383294-673a-40a9-ab9b-6ac98c648a09" containerName="istio-proxy" Nov 02 01:36:54.633295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.633183 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="773a918a-ee3a-4991-bc33-5ac626a65237" containerName="istio-proxy" Nov 02 01:36:54.637277 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.637253 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.659567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.659543 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-41215f-serving-cert\"" Nov 02 01:36:54.689823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.684651 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq"] Nov 02 01:36:54.708583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.708552 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2db8\" (UniqueName: \"kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.708726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.708628 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.708726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.708664 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.708726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.708694 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.833247 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.833574 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2db8\" (UniqueName: \"kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.833688 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.833775 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:54.835595 4552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-serving-cert: secret "maas-api-e2e-inf-a-41215f-serving-cert" not found Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:54.835679 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls podName:ec7cdbca-b490-4061-a4e9-eae28d7958af nodeName:}" failed. No retries permitted until 2026-11-02 01:36:55.335657677 +0000 UTC m=+1315.441446590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls") pod "maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" (UID: "ec7cdbca-b490-4061-a4e9-eae28d7958af") : secret "maas-api-e2e-inf-a-41215f-serving-cert" not found Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.847413 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.849139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.847730 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:54.861166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:54.860405 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2db8\" (UniqueName: \"kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:55.345139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:55.344727 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:55.354148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:55.349177 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\") pod \"maas-api-e2e-inf-a-41215f-666948fd5-vkzlq\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:55.579666 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:55.579633 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:55.871466 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:36:55.871431 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7cdbca_b490_4061_a4e9_eae28d7958af.slice/crio-f3b4ffd915b762e283a550ac37c902017585de8e6ae1db15eacffbfebccec191 WatchSource:0}: Error finding container f3b4ffd915b762e283a550ac37c902017585de8e6ae1db15eacffbfebccec191: Status 404 returned error can't find the container with id f3b4ffd915b762e283a550ac37c902017585de8e6ae1db15eacffbfebccec191 Nov 02 01:36:55.873339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:55.873312 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq"] Nov 02 01:36:56.467555 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:56.466677 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" event={"ID":"ec7cdbca-b490-4061-a4e9-eae28d7958af","Type":"ContainerStarted","Data":"f3b4ffd915b762e283a550ac37c902017585de8e6ae1db15eacffbfebccec191"} Nov 02 01:36:57.492132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:57.491877 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" event={"ID":"ec7cdbca-b490-4061-a4e9-eae28d7958af","Type":"ContainerStarted","Data":"c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357"} Nov 02 01:36:57.493793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:57.493595 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:36:57.519879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:57.519826 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" podStartSLOduration=2.911276953 podStartE2EDuration="3.519810752s" totalImagesPullingTime="608.533799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:57.513094388 +0000 UTC m=+1317.618883313" watchObservedRunningTime="2026-11-02 01:36:57.519810752 +0000 UTC m=+1317.625599666" Nov 02 01:36:57.793459 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:36:57.793363 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:36:58.112188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.099252 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc"] Nov 02 01:36:58.112188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.110986 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc"] Nov 02 01:36:58.112188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.111222 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.123182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.118867 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-00f1da-openshift-default-dockercfg-c5svk\"" Nov 02 01:36:58.167603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.167563 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.167802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.167678 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.167802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.167709 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.167802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.167769 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.167965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.167806 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2wc\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.167965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.167908 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.168293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.168262 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.168456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.168355 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.168456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.168401 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.286832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.286802 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287056 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287177 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287295 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287395 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287465 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287546 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287664 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287652 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2wc\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.287762 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.287752 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.288223 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.288206 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.288616 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.288595 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.290055 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.290009 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.290993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.290964 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.294825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.294100 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.294825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.294356 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.294825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.294696 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.301479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.301451 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.310019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.309855 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2wc\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.463885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.463794 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:36:58.759665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.759626 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc"] Nov 02 01:36:58.760444 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:36:58.760416 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f40f3a_964f_4d39_bde6_e75bc4e136c2.slice/crio-6630f9eb0e407772904a33a4b8f98a45fd94adf6fb178bf9e074714e3319d330 WatchSource:0}: Error finding container 6630f9eb0e407772904a33a4b8f98a45fd94adf6fb178bf9e074714e3319d330: Status 404 returned error can't find the container with id 6630f9eb0e407772904a33a4b8f98a45fd94adf6fb178bf9e074714e3319d330 Nov 02 01:36:58.764723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.764680 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:36:58.764837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.764778 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:36:58.764837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:58.764821 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:36:59.515335 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:59.515294 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" event={"ID":"f7f40f3a-964f-4d39-bde6-e75bc4e136c2","Type":"ContainerStarted","Data":"60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5"} Nov 02 01:36:59.515539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:59.515355 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" event={"ID":"f7f40f3a-964f-4d39-bde6-e75bc4e136c2","Type":"ContainerStarted","Data":"6630f9eb0e407772904a33a4b8f98a45fd94adf6fb178bf9e074714e3319d330"} Nov 02 01:36:59.544337 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:36:59.544280 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podStartSLOduration=1.5442625639999998 podStartE2EDuration="1.544262564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:59.538517816 +0000 UTC m=+1319.644306732" watchObservedRunningTime="2026-11-02 01:36:59.544262564 +0000 UTC m=+1319.650051478" Nov 02 01:37:00.468429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:00.468398 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:37:00.470632 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:00.470604 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:00.470822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:00.470797 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 01:37:01.468490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:01.468455 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:01.469069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:01.469032 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 01:37:02.481411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:02.479802 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:02.481411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:02.479862 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 01:37:03.467510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:03.467473 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:03.467797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:03.467772 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 01:37:04.470741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:04.470256 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:04.470741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:04.470318 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 01:37:04.523004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:04.522976 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:37:05.471166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:05.471132 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:37:05.471639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:05.471469 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:37:05.472427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:05.472406 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:37:08.087602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.087500 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n"] Nov 02 01:37:08.088912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.088872 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" podUID="b630033d-3b64-4ddc-9e83-e69562939b63" containerName="maas-api" containerID="cri-o://c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20" gracePeriod=30 Nov 02 01:37:08.559887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.559848 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:37:08.688065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.685535 4552 generic.go:356] "Generic (PLEG): container finished" podID="b630033d-3b64-4ddc-9e83-e69562939b63" containerID="c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20" exitCode=0 Nov 02 01:37:08.688065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.685662 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" event={"ID":"b630033d-3b64-4ddc-9e83-e69562939b63","Type":"ContainerDied","Data":"c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20"} Nov 02 01:37:08.688065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.685754 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" event={"ID":"b630033d-3b64-4ddc-9e83-e69562939b63","Type":"ContainerDied","Data":"9dee2938895611da8cdc8a4facf6c96f2ad08d112beb24c2dbef5fa1eef1ac01"} Nov 02 01:37:08.688065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.685803 4552 scope.go:122] "RemoveContainer" containerID="c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20" Nov 02 01:37:08.688065 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.685983 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" Nov 02 01:37:08.701707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.701418 4552 scope.go:122] "RemoveContainer" containerID="c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20" Nov 02 01:37:08.701832 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:37:08.701747 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20\": container with ID starting with c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20 not found: ID does not exist" containerID="c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20" Nov 02 01:37:08.701832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.701798 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20"} err="failed to get container status \"c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20\": rpc error: code = NotFound desc = could not find container \"c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20\": container with ID starting with c57fe7c6f269257d6ba4f2afa288ed12c3b66baaae5d89a632ebcd2de5223a20 not found: ID does not exist" Nov 02 01:37:08.763414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.763323 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls\") pod \"b630033d-3b64-4ddc-9e83-e69562939b63\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " Nov 02 01:37:08.763599 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.763516 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs\") pod \"b630033d-3b64-4ddc-9e83-e69562939b63\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " Nov 02 01:37:08.763751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.763731 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc\" (UniqueName: \"kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc\") pod \"b630033d-3b64-4ddc-9e83-e69562939b63\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " Nov 02 01:37:08.763844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.763822 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca\") pod \"b630033d-3b64-4ddc-9e83-e69562939b63\" (UID: \"b630033d-3b64-4ddc-9e83-e69562939b63\") " Nov 02 01:37:08.764970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.764941 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca" pod "b630033d-3b64-4ddc-9e83-e69562939b63" (UID: "b630033d-3b64-4ddc-9e83-e69562939b63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:37:08.765884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.765844 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls" pod "b630033d-3b64-4ddc-9e83-e69562939b63" (UID: "b630033d-3b64-4ddc-9e83-e69562939b63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:37:08.766273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.766249 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs" pod "b630033d-3b64-4ddc-9e83-e69562939b63" (UID: "b630033d-3b64-4ddc-9e83-e69562939b63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:37:08.766679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.766653 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc" pod "b630033d-3b64-4ddc-9e83-e69562939b63" (UID: "b630033d-3b64-4ddc-9e83-e69562939b63"). InnerVolumeSpecName "kube-api-access-7cbcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:37:08.866755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.866715 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbcc\" (UniqueName: \"kubernetes.io/projected/b630033d-3b64-4ddc-9e83-e69562939b63-kube-api-access-7cbcc\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:08.866755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.866748 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b630033d-3b64-4ddc-9e83-e69562939b63-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:08.866755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.866759 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:08.866755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:08.866767 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b630033d-3b64-4ddc-9e83-e69562939b63-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:09.015268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:09.015177 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n"] Nov 02 01:37:09.022935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:09.022887 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n"] Nov 02 01:37:10.050293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:10.050261 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b630033d-3b64-4ddc-9e83-e69562939b63" path="/var/lib/kubelet/pods/b630033d-3b64-4ddc-9e83-e69562939b63/volumes" Nov 02 01:37:11.485718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:11.485682 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50537f4f-6757d99f76-g7c4n" podUID="b630033d-3b64-4ddc-9e83-e69562939b63" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.119:8443/health\": context deadline exceeded" Nov 02 01:37:14.106281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.106251 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74"] Nov 02 01:37:14.389709 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.389625 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg"] Nov 02 01:37:14.390593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.390569 4552 cpu_manager.go:414] "containerMap: removing container" podUID="b630033d-3b64-4ddc-9e83-e69562939b63" containerName="maas-api" Nov 02 01:37:14.390593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.390592 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b630033d-3b64-4ddc-9e83-e69562939b63" containerName="maas-api" Nov 02 01:37:14.390869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.390855 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="b630033d-3b64-4ddc-9e83-e69562939b63" containerName="maas-api" Nov 02 01:37:14.399339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.399315 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.402191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.402164 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg"] Nov 02 01:37:14.405821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.405797 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2589b2\"/\"kube-root-ca.crt\"" Nov 02 01:37:14.406373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.406352 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2589b2\"/\"openshift-service-ca.crt\"" Nov 02 01:37:14.406522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.406400 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2589b2\"/\"test-model-2589b2-kserve-self-signed-certs\"" Nov 02 01:37:14.406522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.406468 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2589b2\"/\"default-dockercfg-ffxq6\"" Nov 02 01:37:14.467768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.467738 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.467968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.467943 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6nv\" (UniqueName: \"kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.468037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.467998 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.468138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.468108 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.468193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.468180 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.468246 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.468234 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.569319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569261 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.569526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569392 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.569526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569429 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.569526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569498 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.569686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569657 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.569884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569855 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6nv\" (UniqueName: \"kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.570039 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569897 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.570039 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.569855 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.570193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.570073 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.572067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.572047 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.572503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.572482 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.582416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.582388 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6nv\" (UniqueName: \"kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv\") pod \"test-model-2589b2-kserve-7974f8c6c-p7svg\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:14.726825 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:14.724425 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:15.087701 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:15.087664 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg"] Nov 02 01:37:15.088521 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:37:15.088497 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01903806_7eda_48ca_8097_c71608b6920f.slice/crio-8153334f61c24940d8ebcce3838d5e660d5de43ecf3e1fc1d7b1e4ac62496130 WatchSource:0}: Error finding container 8153334f61c24940d8ebcce3838d5e660d5de43ecf3e1fc1d7b1e4ac62496130: Status 404 returned error can't find the container with id 8153334f61c24940d8ebcce3838d5e660d5de43ecf3e1fc1d7b1e4ac62496130 Nov 02 01:37:15.774470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:15.774432 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" event={"ID":"01903806-7eda-48ca-8097-c71608b6920f","Type":"ContainerStarted","Data":"f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda"} Nov 02 01:37:15.774967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:15.774481 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" event={"ID":"01903806-7eda-48ca-8097-c71608b6920f","Type":"ContainerStarted","Data":"8153334f61c24940d8ebcce3838d5e660d5de43ecf3e1fc1d7b1e4ac62496130"} Nov 02 01:37:15.798691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:15.798631 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" podStartSLOduration=1.798617768 podStartE2EDuration="1.798617768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:15.797230256 +0000 UTC m=+1335.903019171" watchObservedRunningTime="2026-11-02 01:37:15.798617768 +0000 UTC m=+1335.904406681" Nov 02 01:37:16.725406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:16.725377 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:17.171983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.171883 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt"] Nov 02 01:37:17.178161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.178131 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.224688 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.224659 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bf1e55-serving-cert\"" Nov 02 01:37:17.225059 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.225039 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt"] Nov 02 01:37:17.352165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.350577 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.352165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.350610 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.352165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.350851 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b67ms\" (UniqueName: \"kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.352165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.350881 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.461193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.458431 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67ms\" (UniqueName: \"kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.461193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.458496 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.461193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.458838 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.461193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.458878 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.475183 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.465675 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.476562 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.476535 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.492751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.487721 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.493355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.491255 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67ms\" (UniqueName: \"kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms\") pod \"maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.503665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.503633 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:17.828656 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:17.828635 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt"] Nov 02 01:37:17.864634 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:37:17.864601 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d32b0e1_25be_41dc_a8b2_e86476d4c70d.slice/crio-42c90e0f4796b353cb6000a04778aa22c0dc907a0834db999bd675efe96f6a89 WatchSource:0}: Error finding container 42c90e0f4796b353cb6000a04778aa22c0dc907a0834db999bd675efe96f6a89: Status 404 returned error can't find the container with id 42c90e0f4796b353cb6000a04778aa22c0dc907a0834db999bd675efe96f6a89 Nov 02 01:37:18.828528 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:18.828490 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" event={"ID":"1d32b0e1-25be-41dc-a8b2-e86476d4c70d","Type":"ContainerStarted","Data":"c2896528a6d0a19a8a6c9a816f2e8c3d21914a671678378b23e2ce99c6ec4557"} Nov 02 01:37:18.829076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:18.829050 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" event={"ID":"1d32b0e1-25be-41dc-a8b2-e86476d4c70d","Type":"ContainerStarted","Data":"42c90e0f4796b353cb6000a04778aa22c0dc907a0834db999bd675efe96f6a89"} Nov 02 01:37:18.829265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:18.829249 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:18.857040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:18.856980 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" podStartSLOduration=1.32533568 podStartE2EDuration="1.856964371s" totalImagesPullingTime="531.628691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:18.853322789 +0000 UTC m=+1338.959111703" watchObservedRunningTime="2026-11-02 01:37:18.856964371 +0000 UTC m=+1338.962753286" Nov 02 01:37:24.725176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:24.725140 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:24.738938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:24.738902 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:24.846012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:24.845980 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:37:25.077526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:25.077437 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:37:27.753107 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:37:27.753066 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:37:28.533156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:28.530147 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:37:29.129471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.129426 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" containerID="cri-o://cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f" gracePeriod=45 Nov 02 01:37:29.498911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.498870 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:37:29.581604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.581557 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5\" (UniqueName: \"kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5\") pod \"cc901fb0-87d2-44b1-9745-39bf870271d3\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " Nov 02 01:37:29.581821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.581786 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs\") pod \"cc901fb0-87d2-44b1-9745-39bf870271d3\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " Nov 02 01:37:29.581906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.581874 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache\") pod \"cc901fb0-87d2-44b1-9745-39bf870271d3\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " Nov 02 01:37:29.582019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.581983 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home\") pod \"cc901fb0-87d2-44b1-9745-39bf870271d3\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " Nov 02 01:37:29.582198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.582177 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm\") pod \"cc901fb0-87d2-44b1-9745-39bf870271d3\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " Nov 02 01:37:29.582288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.582225 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir\") pod \"cc901fb0-87d2-44b1-9745-39bf870271d3\" (UID: \"cc901fb0-87d2-44b1-9745-39bf870271d3\") " Nov 02 01:37:29.583337 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.583308 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir" pod "cc901fb0-87d2-44b1-9745-39bf870271d3" (UID: "cc901fb0-87d2-44b1-9745-39bf870271d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:37:29.583945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.583915 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache" pod "cc901fb0-87d2-44b1-9745-39bf870271d3" (UID: "cc901fb0-87d2-44b1-9745-39bf870271d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:37:29.584053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.583942 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home" pod "cc901fb0-87d2-44b1-9745-39bf870271d3" (UID: "cc901fb0-87d2-44b1-9745-39bf870271d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:37:29.589521 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.589494 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs" pod "cc901fb0-87d2-44b1-9745-39bf870271d3" (UID: "cc901fb0-87d2-44b1-9745-39bf870271d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:37:29.589720 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.589699 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5" pod "cc901fb0-87d2-44b1-9745-39bf870271d3" (UID: "cc901fb0-87d2-44b1-9745-39bf870271d3"). InnerVolumeSpecName "kube-api-access-vzrc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:37:29.591691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.591663 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm" pod "cc901fb0-87d2-44b1-9745-39bf870271d3" (UID: "cc901fb0-87d2-44b1-9745-39bf870271d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:37:29.684272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.684240 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:29.684447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.684294 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:29.684447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.684310 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrc5\" (UniqueName: \"kubernetes.io/projected/cc901fb0-87d2-44b1-9745-39bf870271d3-kube-api-access-vzrc5\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:29.684447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.684322 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc901fb0-87d2-44b1-9745-39bf870271d3-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:29.684447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.684333 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:29.684447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:29.684345 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc901fb0-87d2-44b1-9745-39bf870271d3-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:37:30.173957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.173922 4552 generic.go:356] "Generic (PLEG): container finished" podID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerID="cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f" exitCode=0 Nov 02 01:37:30.174409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.173974 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" event={"ID":"cc901fb0-87d2-44b1-9745-39bf870271d3","Type":"ContainerDied","Data":"cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f"} Nov 02 01:37:30.174409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.174009 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" event={"ID":"cc901fb0-87d2-44b1-9745-39bf870271d3","Type":"ContainerDied","Data":"88567f2028e52f683857581023ebed9cdcf0dd15e994f9297ccfc3764680489d"} Nov 02 01:37:30.174409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.174031 4552 scope.go:122] "RemoveContainer" containerID="cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f" Nov 02 01:37:30.174409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.174254 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" Nov 02 01:37:30.192248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.192222 4552 scope.go:122] "RemoveContainer" containerID="cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f" Nov 02 01:37:30.192641 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:37:30.192607 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f\": container with ID starting with cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f not found: ID does not exist" containerID="cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f" Nov 02 01:37:30.192772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.192648 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f"} err="failed to get container status \"cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f\": rpc error: code = NotFound desc = could not find container \"cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f\": container with ID starting with cc786712007894c429acb4ac2f96977ce5c835e1f4e4a2ddb3316568d61d608f not found: ID does not exist" Nov 02 01:37:30.205164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.204746 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74"] Nov 02 01:37:30.211782 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.211755 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74"] Nov 02 01:37:30.468201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:30.458838 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-50537f4f/e2e-lifecycle-model-50537f4f-kserve-5bd67445b9-mrv74" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:37:32.050240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:32.050206 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" path="/var/lib/kubelet/pods/cc901fb0-87d2-44b1-9745-39bf870271d3/volumes" Nov 02 01:37:34.933309 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.933278 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82"] Nov 02 01:37:34.934420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.934391 4552 cpu_manager.go:414] "containerMap: removing container" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" Nov 02 01:37:34.934420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.934413 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" Nov 02 01:37:34.934571 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.934561 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc901fb0-87d2-44b1-9745-39bf870271d3" containerName="main" Nov 02 01:37:34.940263 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.940228 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:34.948530 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.948451 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b34fc0\"/\"kube-root-ca.crt\"" Nov 02 01:37:34.949705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.949684 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b34fc0\"/\"openshift-service-ca.crt\"" Nov 02 01:37:34.949906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.949886 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b34fc0\"/\"test-model-b34fc0-kserve-self-signed-certs\"" Nov 02 01:37:34.949967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.949957 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b34fc0\"/\"default-dockercfg-s7gh9\"" Nov 02 01:37:34.953997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:34.953923 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82"] Nov 02 01:37:35.018687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.018653 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.018862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.018784 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.018974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.018956 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.019011 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.018987 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.019155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.019130 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.019276 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.019224 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8l5\" (UniqueName: \"kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.134156 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8l5\" (UniqueName: \"kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.134408 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.135533 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.135709 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.135731 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.135845 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.136459 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.139572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.139502 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.145325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.140493 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.145325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.144694 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.163141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.160476 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.171141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.169865 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8l5\" (UniqueName: \"kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5\") pod \"test-model-b34fc0-kserve-fb97777c9-qnj82\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.280945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.266671 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:35.640622 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:37:35.640588 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb503526_1cc8_4ba0_bbf2_701c6f37a0db.slice/crio-800bf6ed0d4a0f83a0ebd84883924b3e9a71d4c7f36414d6905674dad2aa4925 WatchSource:0}: Error finding container 800bf6ed0d4a0f83a0ebd84883924b3e9a71d4c7f36414d6905674dad2aa4925: Status 404 returned error can't find the container with id 800bf6ed0d4a0f83a0ebd84883924b3e9a71d4c7f36414d6905674dad2aa4925 Nov 02 01:37:35.643154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:35.643010 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82"] Nov 02 01:37:36.313140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:36.312712 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" event={"ID":"eb503526-1cc8-4ba0-bbf2-701c6f37a0db","Type":"ContainerStarted","Data":"4a04c2f562a5c375aa959283dd0fccb93de5f575335b54cf4e3072afe79d79e8"} Nov 02 01:37:36.313140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:36.312803 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" event={"ID":"eb503526-1cc8-4ba0-bbf2-701c6f37a0db","Type":"ContainerStarted","Data":"800bf6ed0d4a0f83a0ebd84883924b3e9a71d4c7f36414d6905674dad2aa4925"} Nov 02 01:37:37.267344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.267306 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:37.732805 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.732726 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" podStartSLOduration=3.732704967 podStartE2EDuration="3.732704967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:36.338540089 +0000 UTC m=+1356.444329031" watchObservedRunningTime="2026-11-02 01:37:37.732704967 +0000 UTC m=+1357.838493883" Nov 02 01:37:37.739544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.739513 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6"] Nov 02 01:37:37.745208 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.745187 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.751433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.751396 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9a4595\"/\"test-model-9a4595-kserve-self-signed-certs\"" Nov 02 01:37:37.754033 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.754009 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9a4595\"/\"default-dockercfg-mfqhs\"" Nov 02 01:37:37.755758 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.755734 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9a4595\"/\"kube-root-ca.crt\"" Nov 02 01:37:37.756493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.756218 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9a4595\"/\"openshift-service-ca.crt\"" Nov 02 01:37:37.757646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.757238 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6"] Nov 02 01:37:37.799330 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.799294 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4zp\" (UniqueName: \"kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.799481 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.799360 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.799538 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.799519 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.799653 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.799631 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.799813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.799796 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.799905 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.799887 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901011 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901067 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901144 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4zp\" (UniqueName: \"kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901175 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901284 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901362 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901611 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901579 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901618 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.901897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.901877 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.903502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.903478 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.903799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.903781 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:37.913603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:37.913583 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4zp\" (UniqueName: \"kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp\") pod \"test-model-9a4595-kserve-68d6f4986f-2fbn6\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:38.079587 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:38.079488 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:38.404027 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:37:38.403993 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06e19217_f9be_4f7f_b790_605fe7baba8d.slice/crio-b2680f27e56c353c5759ab1ac744a4e20c9a042472f9f1bb5691249970c55ea3 WatchSource:0}: Error finding container b2680f27e56c353c5759ab1ac744a4e20c9a042472f9f1bb5691249970c55ea3: Status 404 returned error can't find the container with id b2680f27e56c353c5759ab1ac744a4e20c9a042472f9f1bb5691249970c55ea3 Nov 02 01:37:38.406447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:38.406421 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6"] Nov 02 01:37:39.339770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:39.339731 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" event={"ID":"06e19217-f9be-4f7f-b790-605fe7baba8d","Type":"ContainerStarted","Data":"7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b"} Nov 02 01:37:39.340238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:39.339782 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" event={"ID":"06e19217-f9be-4f7f-b790-605fe7baba8d","Type":"ContainerStarted","Data":"b2680f27e56c353c5759ab1ac744a4e20c9a042472f9f1bb5691249970c55ea3"} Nov 02 01:37:39.360496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:39.360452 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" podStartSLOduration=2.360437644 podStartE2EDuration="2.360437644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:39.358037268 +0000 UTC m=+1359.463826182" watchObservedRunningTime="2026-11-02 01:37:39.360437644 +0000 UTC m=+1359.466226557" Nov 02 01:37:40.080936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:40.080901 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:45.267312 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:45.267277 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:45.281017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:45.280994 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:45.421074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:45.421048 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:37:48.082811 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.082781 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:48.102741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.102706 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:48.266021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.265985 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph"] Nov 02 01:37:48.280395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.280367 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.289517 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.289490 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f689bc-serving-cert\"" Nov 02 01:37:48.292848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.292819 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph"] Nov 02 01:37:48.437286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.434579 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.437286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.434752 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.437286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.435012 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.437286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.435095 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77g8m\" (UniqueName: \"kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.487271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.487241 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:37:48.536693 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.536662 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.536999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.536768 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.536999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.536881 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77g8m\" (UniqueName: \"kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.537165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.537145 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.543096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.538152 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.551726 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.551686 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.566084 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.565923 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.566299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.566195 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77g8m\" (UniqueName: \"kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m\") pod \"maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.605402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.605360 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:48.902714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:48.902680 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph"] Nov 02 01:37:48.906847 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:37:48.906814 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b586b83_adea_4ef9_8aa2_76c0bc61988c.slice/crio-2c0f1b099b442056b5fc498669f714712f1c752cabacdfe07f320896755ea614 WatchSource:0}: Error finding container 2c0f1b099b442056b5fc498669f714712f1c752cabacdfe07f320896755ea614: Status 404 returned error can't find the container with id 2c0f1b099b442056b5fc498669f714712f1c752cabacdfe07f320896755ea614 Nov 02 01:37:49.486271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:49.486180 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" event={"ID":"8b586b83-adea-4ef9-8aa2-76c0bc61988c","Type":"ContainerStarted","Data":"a07d677a25ae96efab608dae0ad5b07d3aadc0b8547be8ae4a68b5d8ac7ca7a4"} Nov 02 01:37:49.488021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:49.486877 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" event={"ID":"8b586b83-adea-4ef9-8aa2-76c0bc61988c","Type":"ContainerStarted","Data":"2c0f1b099b442056b5fc498669f714712f1c752cabacdfe07f320896755ea614"} Nov 02 01:37:49.508581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:49.508530 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" podStartSLOduration=1.023700092 podStartE2EDuration="1.508515566s" totalImagesPullingTime="484.815474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:49.506334003 +0000 UTC m=+1369.612122928" watchObservedRunningTime="2026-11-02 01:37:49.508515566 +0000 UTC m=+1369.614304480" Nov 02 01:37:50.495029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:50.494998 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:56.508940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:37:56.508911 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:37:57.774202 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:37:57.772847 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:38:15.392571 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.392355 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq"] Nov 02 01:38:15.399432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.399406 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.408025 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.407993 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-41215f\"/\"default-dockercfg-2kcpp\"" Nov 02 01:38:15.408245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.408033 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-41215f\"/\"test-model-41215f-kserve-self-signed-certs\"" Nov 02 01:38:15.409637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.409613 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-41215f\"/\"openshift-service-ca.crt\"" Nov 02 01:38:15.409798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.409778 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-41215f\"/\"kube-root-ca.crt\"" Nov 02 01:38:15.413781 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.413754 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq"] Nov 02 01:38:15.481622 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.481589 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.481865 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.481847 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hpgf\" (UniqueName: \"kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.481927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.481894 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.482035 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.482015 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.482099 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.482055 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.482215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.482191 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584088 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584349 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hpgf\" (UniqueName: \"kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584450 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584412 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584451 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584483 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584516 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584554 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584708 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.584804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.584790 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.587363 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.587334 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.587646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.587628 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.596418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.596396 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hpgf\" (UniqueName: \"kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf\") pod \"test-model-41215f-kserve-7d5b454bb9-b5xqq\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:15.730194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:15.729816 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:16.151731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:16.151676 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq"] Nov 02 01:38:17.090962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:17.090927 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" event={"ID":"2b994055-2f56-4652-978e-da8a9e5927e5","Type":"ContainerStarted","Data":"3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff"} Nov 02 01:38:17.090962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:17.090966 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" event={"ID":"2b994055-2f56-4652-978e-da8a9e5927e5","Type":"ContainerStarted","Data":"2342a2af12482fffc35e4505d522cf0d843284e02004a41c31d0989c3657e2e4"} Nov 02 01:38:17.114037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:17.113978 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" podStartSLOduration=2.113965211 podStartE2EDuration="2.113965211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:17.111612168 +0000 UTC m=+1397.217401082" watchObservedRunningTime="2026-11-02 01:38:17.113965211 +0000 UTC m=+1397.219754125" Nov 02 01:38:17.729930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:17.728183 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:19.777569 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:19.777535 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:38:25.771148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:25.765193 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:25.795275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:25.795233 4552 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:38:25.921156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:25.919668 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:26.059805 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:26.059703 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:38:26.272593 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:26.269482 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 02 01:38:26.475142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:26.467237 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.128:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:38:26.557082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:26.540684 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:38:26.718576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:26.718545 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:38:27.759250 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:38:27.759213 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:38:33.749725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.746658 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9"] Nov 02 01:38:33.763542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.763514 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.775175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.768359 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9"] Nov 02 01:38:33.775175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.770852 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f689bc\"/\"default-dockercfg-crlrt\"" Nov 02 01:38:33.775175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.770951 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f689bc\"/\"test-model-f689bc-kserve-self-signed-certs\"" Nov 02 01:38:33.775175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.771045 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f689bc\"/\"kube-root-ca.crt\"" Nov 02 01:38:33.775175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.771096 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f689bc\"/\"openshift-service-ca.crt\"" Nov 02 01:38:33.803244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.803210 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.803428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.803300 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.803428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.803348 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.803428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.803423 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.803613 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.803539 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.803677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.803626 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6txn\" (UniqueName: \"kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905273 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905352 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905385 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905470 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905635 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905493 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905635 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905548 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6txn\" (UniqueName: \"kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905824 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905945 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.905970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.905961 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.907966 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.907942 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.908149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.908102 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:33.917809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:33.917789 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6txn\" (UniqueName: \"kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn\") pod \"test-model-f689bc-kserve-687fbbf795-fzzz9\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:34.113483 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:34.113392 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:34.482003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:34.481977 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9"] Nov 02 01:38:34.482392 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:38:34.482364 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cf7aa15_4f00_485a_a071_4f7257deca3f.slice/crio-52ed28a9e292d95238a186c74a122aeddb99cf706cb4785498c5140b6a6c1412 WatchSource:0}: Error finding container 52ed28a9e292d95238a186c74a122aeddb99cf706cb4785498c5140b6a6c1412: Status 404 returned error can't find the container with id 52ed28a9e292d95238a186c74a122aeddb99cf706cb4785498c5140b6a6c1412 Nov 02 01:38:34.814907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:34.814805 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" event={"ID":"6cf7aa15-4f00-485a-a071-4f7257deca3f","Type":"ContainerStarted","Data":"155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee"} Nov 02 01:38:34.814907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:34.814860 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" event={"ID":"6cf7aa15-4f00-485a-a071-4f7257deca3f","Type":"ContainerStarted","Data":"52ed28a9e292d95238a186c74a122aeddb99cf706cb4785498c5140b6a6c1412"} Nov 02 01:38:34.835884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:34.835830 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" podStartSLOduration=1.835814344 podStartE2EDuration="1.835814344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:34.833943185 +0000 UTC m=+1414.939732100" watchObservedRunningTime="2026-11-02 01:38:34.835814344 +0000 UTC m=+1414.941603285" Nov 02 01:38:35.114406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:35.114330 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:37.987005 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.986956 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd"] Nov 02 01:38:37.993304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.993279 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:37.994763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.994739 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd"] Nov 02 01:38:37.997252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.997230 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bf1e55\"/\"default-dockercfg-9n5pt\"" Nov 02 01:38:37.997369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.997348 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bf1e55\"/\"test-model-bf1e55-kserve-self-signed-certs\"" Nov 02 01:38:37.998191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.998172 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bf1e55\"/\"openshift-service-ca.crt\"" Nov 02 01:38:37.998191 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:37.998185 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bf1e55\"/\"kube-root-ca.crt\"" Nov 02 01:38:38.101301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.101267 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.101490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.101446 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.101548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.101504 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vq6p\" (UniqueName: \"kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.101612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.101590 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.101698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.101685 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.101900 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.101888 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.211732 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vq6p\" (UniqueName: \"kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.211791 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.211852 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.212104 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.212162 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.212311 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.212799 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.213839 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.216468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.215612 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.222139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.220878 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.262207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.249698 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vq6p\" (UniqueName: \"kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.268141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.267812 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs\") pod \"test-model-bf1e55-kserve-7cf46c9c59-sqvdd\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.326148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.325809 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:38.639828 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:38:38.639791 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66129927_16bf_48e1_835d_f236f06dd6d5.slice/crio-66646784d2c0f908cc84e2845aeda1928cc26103b20a72b114560c5dda4c6924 WatchSource:0}: Error finding container 66646784d2c0f908cc84e2845aeda1928cc26103b20a72b114560c5dda4c6924: Status 404 returned error can't find the container with id 66646784d2c0f908cc84e2845aeda1928cc26103b20a72b114560c5dda4c6924 Nov 02 01:38:38.640783 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.640730 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd"] Nov 02 01:38:38.895037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.894949 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" event={"ID":"66129927-16bf-48e1-835d-f236f06dd6d5","Type":"ContainerStarted","Data":"335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978"} Nov 02 01:38:38.895037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.894989 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" event={"ID":"66129927-16bf-48e1-835d-f236f06dd6d5","Type":"ContainerStarted","Data":"66646784d2c0f908cc84e2845aeda1928cc26103b20a72b114560c5dda4c6924"} Nov 02 01:38:38.919666 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:38.919616 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" podStartSLOduration=1.9196027839999998 podStartE2EDuration="1.919602784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:38.913761519 +0000 UTC m=+1419.019550435" watchObservedRunningTime="2026-11-02 01:38:38.919602784 +0000 UTC m=+1419.025391698" Nov 02 01:38:39.327086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:39.327053 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:40.649859 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.647289 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z"] Nov 02 01:38:40.663874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.658885 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.663874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.661648 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z"] Nov 02 01:38:40.672181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.664256 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8b3bae-serving-cert\"" Nov 02 01:38:40.828141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.828019 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.828357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.828165 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.828357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.828206 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8st4d\" (UniqueName: \"kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.836847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.831674 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.933057 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.932963 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.933296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.933188 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.933296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.933289 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.933423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.933326 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8st4d\" (UniqueName: \"kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.935961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.935935 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.936077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.936035 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.936077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.936040 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.948003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.947977 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8st4d\" (UniqueName: \"kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d\") pod \"maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:40.998646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:40.998609 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:41.359806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:41.359771 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z"] Nov 02 01:38:41.929701 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:41.929662 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" event={"ID":"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae","Type":"ContainerStarted","Data":"6d291af1a273cfc1e3209bc6e7eaaec1bfb5f3f4960a3d581360a4acfeab4a1c"} Nov 02 01:38:42.940728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:42.940689 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" event={"ID":"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae","Type":"ContainerStarted","Data":"b1d2c51f76b126815134f4b5ba1b68a6e40352968ca40cf9fc688e4e3012c231"} Nov 02 01:38:42.941231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:42.940793 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:43.010401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:43.010351 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" podStartSLOduration=2.426589111 podStartE2EDuration="3.010338421s" totalImagesPullingTime="583.74931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:43.005946358 +0000 UTC m=+1423.111735272" watchObservedRunningTime="2026-11-02 01:38:43.010338421 +0000 UTC m=+1423.116127335" Nov 02 01:38:44.118957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:44.118923 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:44.131805 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:44.131780 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:44.972506 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:44.972474 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:38:48.326549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:48.326513 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:48.343432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:48.343404 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:48.958421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:48.958387 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:38:49.013361 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:38:49.013327 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:38:57.753633 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:38:57.753594 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:39:10.972262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:10.972214 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" podUID="6cf7aa15-4f00-485a-a071-4f7257deca3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.062169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.061344 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.066147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.002465 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.133:8000/ready\": context deadline exceeded" Nov 02 01:39:11.276196 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.274373 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 02 01:39:11.480182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.464394 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.538199 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.485692 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.731457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.705195 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" podUID="2b994055-2f56-4652-978e-da8a9e5927e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.731457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.724337 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.865759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:11.865648 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.071328 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.071270 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.195551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.192204 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.297929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.296352 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:39:12.312393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.312357 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 02 01:39:12.314745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.314721 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:39:12.388808 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.388782 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:39:12.402882 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:12.402853 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:39:27.753513 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:39:27.753468 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:39:41.259204 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:39:41.258966 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:39:57.754915 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:39:57.754868 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:40:27.757967 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:40:27.757924 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:40:40.174695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:40.174660 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn"] Nov 02 01:40:40.175187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:40.174906 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" containerID="cri-o://4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990" gracePeriod=30 Nov 02 01:40:45.488179 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.488149 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:40:45.517912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.517883 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518092 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.517974 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518103 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518148 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518200 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518272 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518314 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518521 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.518628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.518592 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data\") pod \"a3943a20-f352-4b03-bb3b-402d97c5e103\" (UID: \"a3943a20-f352-4b03-bb3b-402d97c5e103\") " Nov 02 01:40:45.519819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519156 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:40:45.519819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519218 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:40:45.519819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519502 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:40:45.519819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519791 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:40:45.520290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519888 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3943a20-f352-4b03-bb3b-402d97c5e103-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.520290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519908 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.520290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.519924 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.520449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.520328 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:40:45.522981 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.522946 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:40:45.523983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.523949 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:40:45.526951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.526258 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "kube-api-access-lvzfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:40:45.526951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.526367 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy" pod "a3943a20-f352-4b03-bb3b-402d97c5e103" (UID: "a3943a20-f352-4b03-bb3b-402d97c5e103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:40:45.530743 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.530718 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf"] Nov 02 01:40:45.531664 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.531646 4552 cpu_manager.go:414] "containerMap: removing container" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" Nov 02 01:40:45.531664 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.531664 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" Nov 02 01:40:45.531856 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.531845 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerName="istio-proxy" Nov 02 01:40:45.536312 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.536292 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.548013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.547991 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bf1e55-openshift-default-dockercfg-w8kcb\"" Nov 02 01:40:45.566217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.566189 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf"] Nov 02 01:40:45.621063 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621031 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621063 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621064 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621081 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621269 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621469 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621447 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621487 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621566 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621625 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmlz\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621683 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.621824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621812 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvzfq\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-kube-api-access-lvzfq\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.621869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621827 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.621869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621838 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.621869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621848 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.621869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621857 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.621869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.621865 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3943a20-f352-4b03-bb3b-402d97c5e103-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:40:45.727578 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.726849 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.727578 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727290 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmlz\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.727578 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727374 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.727903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727667 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.727903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727694 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.727903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727717 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.727903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727776 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.728110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727946 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.728110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.727987 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.728110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.728018 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.728466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.728444 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.728558 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.728474 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.728986 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.728966 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.729078 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.729034 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.733224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.731915 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.734041 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.734016 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.739956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.739928 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.740507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.740485 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmlz\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.880508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.880471 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:45.981222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.981097 4552 generic.go:356] "Generic (PLEG): container finished" podID="a3943a20-f352-4b03-bb3b-402d97c5e103" containerID="4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990" exitCode=0 Nov 02 01:40:45.981391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.981200 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" Nov 02 01:40:45.981391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.981326 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" event={"ID":"a3943a20-f352-4b03-bb3b-402d97c5e103","Type":"ContainerDied","Data":"4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990"} Nov 02 01:40:45.981507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.981414 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn" event={"ID":"a3943a20-f352-4b03-bb3b-402d97c5e103","Type":"ContainerDied","Data":"1ad15dbf51855383773b6795d10561bea0ac75f253cca380325bc23bd284aea9"} Nov 02 01:40:45.981507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.981449 4552 scope.go:122] "RemoveContainer" containerID="4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990" Nov 02 01:40:45.999597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.999554 4552 scope.go:122] "RemoveContainer" containerID="4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990" Nov 02 01:40:45.999908 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:40:45.999884 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990\": container with ID starting with 4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990 not found: ID does not exist" containerID="4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990" Nov 02 01:40:45.999994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:45.999919 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990"} err="failed to get container status \"4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990\": rpc error: code = NotFound desc = could not find container \"4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990\": container with ID starting with 4aad41405e24cc95266bbfe3e55057a435c901189b2eb0a7a6eb10de5bda9990 not found: ID does not exist" Nov 02 01:40:46.014603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.014556 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn"] Nov 02 01:40:46.020377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.020355 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wfcnn"] Nov 02 01:40:46.049266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.049242 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3943a20-f352-4b03-bb3b-402d97c5e103" path="/var/lib/kubelet/pods/a3943a20-f352-4b03-bb3b-402d97c5e103/volumes" Nov 02 01:40:46.173710 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:40:46.173680 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835e5898_3c65_480d_8b5c_917f255de079.slice/crio-6583f9f4b0a7defd6fefef9c81415ddde648e4196259df76b40270f8b62cb29c WatchSource:0}: Error finding container 6583f9f4b0a7defd6fefef9c81415ddde648e4196259df76b40270f8b62cb29c: Status 404 returned error can't find the container with id 6583f9f4b0a7defd6fefef9c81415ddde648e4196259df76b40270f8b62cb29c Nov 02 01:40:46.174655 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.174633 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf"] Nov 02 01:40:46.176864 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.176829 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:40:46.176941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.176897 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:40:46.176941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.176927 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:40:46.994441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.994388 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" event={"ID":"835e5898-3c65-480d-8b5c-917f255de079","Type":"ContainerStarted","Data":"0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a"} Nov 02 01:40:46.994924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:46.994456 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" event={"ID":"835e5898-3c65-480d-8b5c-917f255de079","Type":"ContainerStarted","Data":"6583f9f4b0a7defd6fefef9c81415ddde648e4196259df76b40270f8b62cb29c"} Nov 02 01:40:47.028134 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:47.028055 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podStartSLOduration=85.02804072 podStartE2EDuration="1m25.02804072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:47.020976127 +0000 UTC m=+1547.126765042" watchObservedRunningTime="2026-11-02 01:40:47.02804072 +0000 UTC m=+1547.133829633" Nov 02 01:40:47.881216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:47.881177 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:47.883850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:47.883815 4552 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 01:40:47.884029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:47.883879 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 01:40:48.886740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:48.886704 4552 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 01:40:48.887436 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:48.887401 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 01:40:49.881478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:49.881433 4552 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 01:40:49.881688 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:49.881488 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 01:40:50.881345 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:50.881309 4552 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 01:40:50.881773 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:50.881368 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 01:40:51.881279 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:51.881234 4552 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 01:40:51.881494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:51.881311 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 01:40:52.930823 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:52.930792 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:52.931606 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:52.931585 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:52.933584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:52.933567 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:40:55.671914 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:55.671870 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq"] Nov 02 01:40:55.672463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:40:55.672078 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" containerID="cri-o://484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e" gracePeriod=30 Nov 02 01:40:57.752574 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:40:57.752539 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:41:01.070783 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.070757 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:41:01.109269 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.109230 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz"] Nov 02 01:41:01.110482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.110457 4552 cpu_manager.go:414] "containerMap: removing container" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" Nov 02 01:41:01.110482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.110483 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" Nov 02 01:41:01.110754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.110739 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerName="istio-proxy" Nov 02 01:41:01.114562 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.114542 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.119512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.119489 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f689bc-openshift-default-dockercfg-zlxb6\"" Nov 02 01:41:01.133876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.129588 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.133876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.133185 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz"] Nov 02 01:41:01.133876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.133643 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:01.134329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134306 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.134411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134383 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.134466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134436 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.134524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134495 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.134576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134541 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.134647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134633 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134740 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.134806 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token\") pod \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\" (UID: \"5c247f7d-fd19-4518-9f7c-fb64606ba2dd\") " Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.135002 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.135471 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.135960 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.135985 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.136000 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.137400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.136201 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:01.139130 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.138363 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:01.140166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.140139 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:01.142491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.142466 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:01.151168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.151131 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "kube-api-access-6c9ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:01.153252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.153228 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token" pod "5c247f7d-fd19-4518-9f7c-fb64606ba2dd" (UID: "5c247f7d-fd19-4518-9f7c-fb64606ba2dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:01.192425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.190637 4552 generic.go:356] "Generic (PLEG): container finished" podID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" containerID="484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e" exitCode=0 Nov 02 01:41:01.192425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.190681 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" event={"ID":"5c247f7d-fd19-4518-9f7c-fb64606ba2dd","Type":"ContainerDied","Data":"484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e"} Nov 02 01:41:01.192425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.190731 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" event={"ID":"5c247f7d-fd19-4518-9f7c-fb64606ba2dd","Type":"ContainerDied","Data":"70499b79d2598701f0b7d819f2bd615bc42d4cf128415a9feb3c6e22e64b01ea"} Nov 02 01:41:01.192425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.190760 4552 scope.go:122] "RemoveContainer" containerID="484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e" Nov 02 01:41:01.192425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.190950 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq" Nov 02 01:41:01.213415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.211657 4552 scope.go:122] "RemoveContainer" containerID="484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e" Nov 02 01:41:01.219937 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:41:01.219884 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e\": container with ID starting with 484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e not found: ID does not exist" containerID="484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e" Nov 02 01:41:01.220169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.219931 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e"} err="failed to get container status \"484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e\": rpc error: code = NotFound desc = could not find container \"484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e\": container with ID starting with 484717522d0c73884177775fd2162e78d3a6bd9bad9426716a3b889aee33078e not found: ID does not exist" Nov 02 01:41:01.230846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.230821 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq"] Nov 02 01:41:01.236607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.236583 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-s89qq"] Nov 02 01:41:01.237463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237434 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.237604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237481 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.237604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237566 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.237720 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237670 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.237813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237793 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.237949 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237927 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh999\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.238010 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.237973 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.238063 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238038 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.238284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238267 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.238417 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238404 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.238472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238424 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.238472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238439 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.238472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238453 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9ks\" (UniqueName: \"kubernetes.io/projected/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-kube-api-access-6c9ks\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.238472 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238469 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.238669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.238483 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c247f7d-fd19-4518-9f7c-fb64606ba2dd-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:01.340249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340160 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh999\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.340249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340226 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.340470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340259 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.340527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340490 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.340584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340540 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.340639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340589 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340707 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340757 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.340856 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.341003 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.341137 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.341447 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.341505 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.341579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.341513 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.343250 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.343228 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.343754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.343735 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.353576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.353551 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.353794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.353765 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh999\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.446086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.446038 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:01.750583 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:41:01.750550 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4d4da0_8e13_4dec_82ab_1fd2dbb0d552.slice/crio-64fef0833413b0e09bb0e035217b316dd00f03aafdc43b8560f60fe018d86602 WatchSource:0}: Error finding container 64fef0833413b0e09bb0e035217b316dd00f03aafdc43b8560f60fe018d86602: Status 404 returned error can't find the container with id 64fef0833413b0e09bb0e035217b316dd00f03aafdc43b8560f60fe018d86602 Nov 02 01:41:01.753897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.753866 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz"] Nov 02 01:41:01.756419 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.756366 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:01.756782 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.756756 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:01.757135 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:01.757092 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:02.055426 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.055343 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c247f7d-fd19-4518-9f7c-fb64606ba2dd" path="/var/lib/kubelet/pods/5c247f7d-fd19-4518-9f7c-fb64606ba2dd/volumes" Nov 02 01:41:02.203589 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.203543 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" event={"ID":"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552","Type":"ContainerStarted","Data":"d11056c596f14bd73e52c222653bdf3d503748b4fee140b0ea3c31d2d48a37af"} Nov 02 01:41:02.204087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.203613 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" event={"ID":"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552","Type":"ContainerStarted","Data":"64fef0833413b0e09bb0e035217b316dd00f03aafdc43b8560f60fe018d86602"} Nov 02 01:41:02.230965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.230915 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podStartSLOduration=84.230897909 podStartE2EDuration="1m24.230897909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:02.227607336 +0000 UTC m=+1562.333396250" watchObservedRunningTime="2026-11-02 01:41:02.230897909 +0000 UTC m=+1562.336686824" Nov 02 01:41:02.446374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.446286 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:02.448184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.448093 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:02.448355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:02.448184 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:03.448916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:03.448883 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:03.449549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:03.449517 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:04.467144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:04.460502 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:04.467144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:04.460573 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:05.446492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:05.446457 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:05.446775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:05.446749 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:06.449145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:06.447470 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:06.449145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:06.447529 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:07.453152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:07.452633 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:07.453152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:07.452710 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:08.447370 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:08.447322 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:41:08.447585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:08.447389 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 01:41:09.451999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:09.451910 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:09.452505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:09.452480 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:09.453271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:09.453245 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:41:10.475222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:10.475181 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t"] Nov 02 01:41:10.475666 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:10.475491 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" containerID="cri-o://d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049" gracePeriod=30 Nov 02 01:41:13.058540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:13.058496 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:41:15.880186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.874521 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:41:15.919508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.919475 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89"] Nov 02 01:41:15.920442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.920415 4552 cpu_manager.go:414] "containerMap: removing container" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" Nov 02 01:41:15.920442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.920436 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" Nov 02 01:41:15.920690 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.920667 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerName="istio-proxy" Nov 02 01:41:15.924985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.924962 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:15.941807 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.941771 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89"] Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954173 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954240 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954288 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954340 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954512 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954699 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954746 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954850 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.955022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.954923 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token\") pod \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\" (UID: \"6ac3ae08-3b39-4d9a-952b-c859af24c6a9\") " Nov 02 01:41:15.956842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.955966 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:15.956842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.956284 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:15.956842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.956470 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:15.956842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.956809 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:15.956842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.956750 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:15.958493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.958457 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:15.959729 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.959694 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:15.960926 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.960895 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "kube-api-access-tfnb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:15.968318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:15.968291 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo" pod "6ac3ae08-3b39-4d9a-952b-c859af24c6a9" (UID: "6ac3ae08-3b39-4d9a-952b-c859af24c6a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:16.057618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.057560 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.057786 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.057720 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.057848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.057798 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmw9\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.057928 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.057908 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.057983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.057949 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.058083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058063 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.058273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058256 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.058377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058359 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.058417 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058399 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.058577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058562 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058583 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058600 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnb5\" (UniqueName: \"kubernetes.io/projected/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-kube-api-access-tfnb5\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058615 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058629 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058643 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058658 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058672 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.058913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.058685 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac3ae08-3b39-4d9a-952b-c859af24c6a9-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:16.160714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.160636 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.160902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.160745 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.160902 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.160787 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161008 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.160945 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161099 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161074 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161153 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmw9\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161197 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161234 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161283 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161316 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161381 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161680 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161837 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.161931 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.161837 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.163455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.163434 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.163772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.163758 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.173490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.173462 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.173765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.173748 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmw9\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.256416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.256384 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:16.382314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.382280 4552 generic.go:356] "Generic (PLEG): container finished" podID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" containerID="d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049" exitCode=0 Nov 02 01:41:16.382572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.382556 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" event={"ID":"6ac3ae08-3b39-4d9a-952b-c859af24c6a9","Type":"ContainerDied","Data":"d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049"} Nov 02 01:41:16.382641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.382600 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" event={"ID":"6ac3ae08-3b39-4d9a-952b-c859af24c6a9","Type":"ContainerDied","Data":"716965bed473e69c1afddd075d036144c888f46cc8469c35cfc1ef3ddd55edb7"} Nov 02 01:41:16.382641 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.382624 4552 scope.go:122] "RemoveContainer" containerID="d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049" Nov 02 01:41:16.385403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.382762 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t" Nov 02 01:41:16.410397 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.406720 4552 scope.go:122] "RemoveContainer" containerID="d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049" Nov 02 01:41:16.410397 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:41:16.407705 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049\": container with ID starting with d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049 not found: ID does not exist" containerID="d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049" Nov 02 01:41:16.410397 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.407766 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049"} err="failed to get container status \"d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049\": rpc error: code = NotFound desc = could not find container \"d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049\": container with ID starting with d7d83d32e91d008f048ee3dbe322f2452ff49402232461cd5c8b982a51beb049 not found: ID does not exist" Nov 02 01:41:16.411020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.410966 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t"] Nov 02 01:41:16.416227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.416191 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gn44t"] Nov 02 01:41:16.561173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.559642 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89"] Nov 02 01:41:16.575670 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.575616 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:16.575873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.575721 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:16.575873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:16.575768 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:17.491167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:17.491075 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" event={"ID":"c9560f4a-a1ac-49c2-9829-728c386c6260","Type":"ContainerStarted","Data":"8443997b7217ef0f4c0572281d24cbffc76dc202277414cb5e12450755bef2ee"} Nov 02 01:41:17.491167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:17.491155 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" event={"ID":"c9560f4a-a1ac-49c2-9829-728c386c6260","Type":"ContainerStarted","Data":"5de347349e2429124c9f10dd8024be0c088e30248bdf700f3046b570d38ab9b8"} Nov 02 01:41:17.525393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:17.525334 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podStartSLOduration=99.525320423 podStartE2EDuration="1m39.525320423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:17.521144121 +0000 UTC m=+1577.626933032" watchObservedRunningTime="2026-11-02 01:41:17.525320423 +0000 UTC m=+1577.631109336" Nov 02 01:41:18.087786 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:18.087750 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac3ae08-3b39-4d9a-952b-c859af24c6a9" path="/var/lib/kubelet/pods/6ac3ae08-3b39-4d9a-952b-c859af24c6a9/volumes" Nov 02 01:41:18.275145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:18.274867 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:18.278147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:18.276730 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:18.278147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:18.276788 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 01:41:19.269147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:19.257464 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:19.269147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:19.257524 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 01:41:20.263144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:20.259234 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:20.263144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:20.259307 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 01:41:21.260145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:21.259694 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:21.260145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:21.259753 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 01:41:22.266140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:22.265484 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:22.266140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:22.265542 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 02 01:41:23.275647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:23.275607 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:23.276954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:23.276929 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:23.279633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:23.279611 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:41:25.212279 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:25.212241 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz"] Nov 02 01:41:25.212690 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:25.212467 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" containerID="cri-o://d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce" gracePeriod=30 Nov 02 01:41:25.287698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:25.287662 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd"] Nov 02 01:41:25.287967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:25.287939 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" containerID="cri-o://194bc8dc5bb761a7f03a33d633c4f6cc025f5729bb9c205231e019d3982b02ea" gracePeriod=30 Nov 02 01:41:27.753457 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:41:27.753409 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:41:30.632888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.632860 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:41:30.682379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.682336 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s"] Nov 02 01:41:30.683852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.683824 4552 cpu_manager.go:414] "containerMap: removing container" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" Nov 02 01:41:30.683852 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.683850 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" Nov 02 01:41:30.684111 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.684096 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerName="istio-proxy" Nov 02 01:41:30.689519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.689494 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.702627 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.695904 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-edef28-openshift-default-dockercfg-bg4zb\"" Nov 02 01:41:30.702627 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.700382 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s"] Nov 02 01:41:30.720136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.720090 4552 generic.go:356] "Generic (PLEG): container finished" podID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerID="194bc8dc5bb761a7f03a33d633c4f6cc025f5729bb9c205231e019d3982b02ea" exitCode=0 Nov 02 01:41:30.720351 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.720324 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" event={"ID":"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae","Type":"ContainerDied","Data":"194bc8dc5bb761a7f03a33d633c4f6cc025f5729bb9c205231e019d3982b02ea"} Nov 02 01:41:30.722471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.722448 4552 generic.go:356] "Generic (PLEG): container finished" podID="3409c2a9-1820-4a11-95cd-b05b011a023f" containerID="d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce" exitCode=0 Nov 02 01:41:30.722568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.722491 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" event={"ID":"3409c2a9-1820-4a11-95cd-b05b011a023f","Type":"ContainerDied","Data":"d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce"} Nov 02 01:41:30.722568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.722523 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" Nov 02 01:41:30.722568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.722534 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz" event={"ID":"3409c2a9-1820-4a11-95cd-b05b011a023f","Type":"ContainerDied","Data":"1e18e77aada980184d2f85e92a241d9e862a406d17eba1c5e73676cef788ee70"} Nov 02 01:41:30.722568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.722565 4552 scope.go:122] "RemoveContainer" containerID="d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce" Nov 02 01:41:30.745561 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.745310 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:41:30.745887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.745869 4552 scope.go:122] "RemoveContainer" containerID="d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce" Nov 02 01:41:30.748388 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:41:30.748359 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce\": container with ID starting with d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce not found: ID does not exist" containerID="d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce" Nov 02 01:41:30.748850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.748822 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce"} err="failed to get container status \"d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce\": rpc error: code = NotFound desc = could not find container \"d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce\": container with ID starting with d18bdcb8f99838803c45ee0ce556ba22ff601ffc0a2a7f90db4650a2a9b7bdce not found: ID does not exist" Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775677 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775732 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775784 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775824 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775887 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775925 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.775993 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.778157 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.776063 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.779595 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert\") pod \"3409c2a9-1820-4a11-95cd-b05b011a023f\" (UID: \"3409c2a9-1820-4a11-95cd-b05b011a023f\") " Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780066 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780111 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780236 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780278 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780687 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780724 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dqz\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780751 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780872 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.780899 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.782401 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.783010 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.783705 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.783906 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.784142 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.794535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.792526 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.796877 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.796823 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:30.797888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.797858 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:30.799544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.799519 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4" pod "3409c2a9-1820-4a11-95cd-b05b011a023f" (UID: "3409c2a9-1820-4a11-95cd-b05b011a023f"). InnerVolumeSpecName "kube-api-access-9fcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:30.807767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.807744 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87"] Nov 02 01:41:30.808771 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.808750 4552 cpu_manager.go:414] "containerMap: removing container" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" Nov 02 01:41:30.808881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.808773 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" Nov 02 01:41:30.808956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.808942 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" containerName="istio-proxy" Nov 02 01:41:30.815411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.815393 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.832690 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.832626 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87"] Nov 02 01:41:30.882412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882322 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882416 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882467 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882531 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882579 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882624 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.882799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882656 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882798 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.882911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882827 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.883005 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.882954 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket\") pod \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\" (UID: \"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae\") " Nov 02 01:41:30.883233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.883209 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.884492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884474 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.884632 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884622 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.884716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884707 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.884786 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884777 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.884888 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884878 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxhd\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.884995 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884986 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.885080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.885072 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.885194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.885179 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.885313 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.883211 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.885313 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.883822 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.885429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884011 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.885429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.884106 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:30.885798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.885768 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.885919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.885897 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.885993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.885941 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dqz\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.885993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.885970 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.886100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886008 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.886100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886037 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.886228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886146 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.886228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886175 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.886333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886268 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.886474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886444 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.886542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886522 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.886646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886621 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.886767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886744 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.887012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.886991 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.887154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887103 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887141 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887152 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887163 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887172 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887182 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887190 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887198 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887207 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcx4\" (UniqueName: \"kubernetes.io/projected/3409c2a9-1820-4a11-95cd-b05b011a023f-kube-api-access-9fcx4\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887216 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887226 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3409c2a9-1820-4a11-95cd-b05b011a023f-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887235 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887243 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3409c2a9-1820-4a11-95cd-b05b011a023f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:30.887824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.887524 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.888511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.888470 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:30.888632 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.888612 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:30.888695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.888652 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:30.890068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.890052 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.890283 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.890238 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4" pod "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" (UID: "5ff388f3-860c-4567-9cf4-bab4d4ebe2ae"). InnerVolumeSpecName "kube-api-access-l5nr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:30.890390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.890374 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.899444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.899418 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dqz\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.899584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.899567 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.988893 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.988994 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.989078 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxhd\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.989201 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.989255 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.989281 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.990871 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.991241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.990871 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:30.995677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.994695 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996418 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996551 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996577 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996823 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5nr4\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-kube-api-access-l5nr4\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996844 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996859 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:30.996874 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.002414 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.004178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.002738 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.008172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.005599 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.008172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.006423 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxhd\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.008172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.006687 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.009368 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.009342 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.020703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.020676 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:31.057076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.057044 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz"] Nov 02 01:41:31.063679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.063652 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-f6pqz"] Nov 02 01:41:31.136184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.136055 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:31.335991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.335958 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s"] Nov 02 01:41:31.338810 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.338762 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:31.338941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.338841 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:31.338941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.338879 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:31.457932 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:41:31.456791 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f84d5a_e446_4384_bbf9_b800146d4cb1.slice/crio-1ea9bb3035e05319caef4c5960dd13d08804910937d629e6338aec22aaadfebc WatchSource:0}: Error finding container 1ea9bb3035e05319caef4c5960dd13d08804910937d629e6338aec22aaadfebc: Status 404 returned error can't find the container with id 1ea9bb3035e05319caef4c5960dd13d08804910937d629e6338aec22aaadfebc Nov 02 01:41:31.457932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.456855 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87"] Nov 02 01:41:31.460804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.460759 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:31.460881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.460838 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:31.460881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.460866 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:41:31.735165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.735080 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" event={"ID":"e3bb7bfa-d628-46aa-9b3e-19679afd118a","Type":"ContainerStarted","Data":"7104ace13d72dc228892c6ee6b4dc24a3b13b26be5d68204ce0368bc989d7e75"} Nov 02 01:41:31.735165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.735153 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" event={"ID":"e3bb7bfa-d628-46aa-9b3e-19679afd118a","Type":"ContainerStarted","Data":"05d8a8a847e53561aba8ffdd112ddb60d8a5cdc1cb6a18fefa5b3e47f0b47188"} Nov 02 01:41:31.737984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.737916 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" event={"ID":"5ff388f3-860c-4567-9cf4-bab4d4ebe2ae","Type":"ContainerDied","Data":"5674ad33d3004e86cff78525b509760484d2feaab77cb8047a7390c7b572056a"} Nov 02 01:41:31.737984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.737979 4552 scope.go:122] "RemoveContainer" containerID="194bc8dc5bb761a7f03a33d633c4f6cc025f5729bb9c205231e019d3982b02ea" Nov 02 01:41:31.741586 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.738722 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd" Nov 02 01:41:31.796222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.767685 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podStartSLOduration=110.767663053 podStartE2EDuration="1m50.767663053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:31.764908579 +0000 UTC m=+1591.870697497" watchObservedRunningTime="2026-11-02 01:41:31.767663053 +0000 UTC m=+1591.873451968" Nov 02 01:41:31.864453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.863095 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" event={"ID":"03f84d5a-e446-4384-bbf9-b800146d4cb1","Type":"ContainerStarted","Data":"ad33bbef03fbce5880ef4d9c62fc1606e9b990fbf765f8898d1df4d7e680d155"} Nov 02 01:41:31.864453 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.863228 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" event={"ID":"03f84d5a-e446-4384-bbf9-b800146d4cb1","Type":"ContainerStarted","Data":"1ea9bb3035e05319caef4c5960dd13d08804910937d629e6338aec22aaadfebc"} Nov 02 01:41:31.899520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.899463 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podStartSLOduration=110.899444103 podStartE2EDuration="1m50.899444103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:31.893789791 +0000 UTC m=+1591.999578706" watchObservedRunningTime="2026-11-02 01:41:31.899444103 +0000 UTC m=+1592.005233018" Nov 02 01:41:31.916982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.916949 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd"] Nov 02 01:41:31.921980 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:31.921937 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-s27fd"] Nov 02 01:41:32.021373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.021296 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:32.022896 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.022867 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:32.023015 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.022927 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:32.054187 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.054111 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3409c2a9-1820-4a11-95cd-b05b011a023f" path="/var/lib/kubelet/pods/3409c2a9-1820-4a11-95cd-b05b011a023f/volumes" Nov 02 01:41:32.054917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.054879 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff388f3-860c-4567-9cf4-bab4d4ebe2ae" path="/var/lib/kubelet/pods/5ff388f3-860c-4567-9cf4-bab4d4ebe2ae/volumes" Nov 02 01:41:32.141348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.141311 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:32.142919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.142886 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:32.143074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:32.142940 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:33.021222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:33.021189 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:33.021738 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:33.021247 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:33.137032 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:33.137000 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:33.137317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:33.137050 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:34.021311 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:34.021269 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:34.021710 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:34.021328 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:34.141244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:34.141208 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:34.141501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:34.141473 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:35.021919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:35.021889 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:35.022387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:35.022250 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:35.136989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:35.136782 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:35.136989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:35.136905 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:36.021432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:36.021403 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:36.021727 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:36.021696 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:36.145142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:36.137409 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:36.145142 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:36.137463 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:37.021245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:37.021212 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:37.021506 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:37.021481 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:37.137094 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:37.137058 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:37.137355 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:37.137164 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:38.026143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:38.022232 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:38.026143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:38.022293 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 01:41:38.141096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:38.141066 4552 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:38.141427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:38.141394 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 01:41:39.027175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:39.027145 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:39.027679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:39.027664 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:39.028464 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:39.028445 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:41:39.187401 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:39.187331 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:39.190209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:39.189389 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:39.206277 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:41:39.204635 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:41:57.753525 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:41:57.753486 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:42:27.814974 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:42:27.814844 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:42:36.721662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:36.721625 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:42:37.193588 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.193473 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z"] Nov 02 01:42:37.193832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.193744 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" podUID="24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" containerName="maas-api" containerID="cri-o://b1d2c51f76b126815134f4b5ba1b68a6e40352968ca40cf9fc688e4e3012c231" gracePeriod=30 Nov 02 01:42:37.759144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.758767 4552 generic.go:356] "Generic (PLEG): container finished" podID="24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" containerID="b1d2c51f76b126815134f4b5ba1b68a6e40352968ca40cf9fc688e4e3012c231" exitCode=0 Nov 02 01:42:37.759144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.759065 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" event={"ID":"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae","Type":"ContainerDied","Data":"b1d2c51f76b126815134f4b5ba1b68a6e40352968ca40cf9fc688e4e3012c231"} Nov 02 01:42:37.850496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.850406 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:42:37.895959 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.895921 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh"] Nov 02 01:42:37.907257 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.907228 4552 cpu_manager.go:414] "containerMap: removing container" podUID="24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" containerName="maas-api" Nov 02 01:42:37.907476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.907460 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" containerName="maas-api" Nov 02 01:42:37.908087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.908065 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" containerName="maas-api" Nov 02 01:42:37.918460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.918429 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh"] Nov 02 01:42:37.918639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:37.918564 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.002683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.002648 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d\" (UniqueName: \"kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d\") pod \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " Nov 02 01:42:38.002884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.002759 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs\") pod \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " Nov 02 01:42:38.003086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003064 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca\") pod \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " Nov 02 01:42:38.003281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003256 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls\") pod \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\" (UID: \"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae\") " Nov 02 01:42:38.003514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003484 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca" pod "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" (UID: "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:38.003595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003498 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.003654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003604 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.003938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003911 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003954 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.003984 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.004214 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.004272 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.004476 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.004541 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv78s\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.004682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.004665 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:38.005451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.005421 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs" pod "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" (UID: "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:42:38.005451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.005430 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d" pod "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" (UID: "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae"). InnerVolumeSpecName "kube-api-access-8st4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:38.005682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.005660 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls" pod "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" (UID: "24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:42:38.105755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.105704 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.105755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.105762 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106035 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.105790 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106035 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.105872 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106035 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.105916 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106176 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106198 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106253 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106316 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv78s\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106383 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106366 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106409 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106495 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106454 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106594 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:38.106673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106642 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.106725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106671 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8st4d\" (UniqueName: \"kubernetes.io/projected/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-kube-api-access-8st4d\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:38.106725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106693 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:38.106878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.106857 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.108554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.108532 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.108954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.108938 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.122563 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.122524 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.122756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.122643 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv78s\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.249957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.243947 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:38.774104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.774069 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" Nov 02 01:42:38.774104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.774088 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z" event={"ID":"24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae","Type":"ContainerDied","Data":"6d291af1a273cfc1e3209bc6e7eaaec1bfb5f3f4960a3d581360a4acfeab4a1c"} Nov 02 01:42:38.774716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.774197 4552 scope.go:122] "RemoveContainer" containerID="b1d2c51f76b126815134f4b5ba1b68a6e40352968ca40cf9fc688e4e3012c231" Nov 02 01:42:38.805339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.805275 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z"] Nov 02 01:42:38.812424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.812392 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8b3bae-f5bc6569-7646z"] Nov 02 01:42:38.899872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.899839 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh"] Nov 02 01:42:38.906288 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:42:38.906253 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6519d4_abe0_4ce9_bc38_38993a7ad791.slice/crio-3a535aca9219d6fb0f7c81ea0e12eb91ea9581a7f812b0d2b43f5164c21c7107 WatchSource:0}: Error finding container 3a535aca9219d6fb0f7c81ea0e12eb91ea9581a7f812b0d2b43f5164c21c7107: Status 404 returned error can't find the container with id 3a535aca9219d6fb0f7c81ea0e12eb91ea9581a7f812b0d2b43f5164c21c7107 Nov 02 01:42:38.909597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.909554 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:42:38.909705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.909657 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:42:38.909766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:38.909706 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:42:39.787806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:39.787751 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" event={"ID":"6f6519d4-abe0-4ce9-bc38-38993a7ad791","Type":"ContainerStarted","Data":"d43fe3baddcd96c4d2b6907d29c23790e5007c79a8d27a4dea3b8e204fcdc8af"} Nov 02 01:42:39.788437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:39.787824 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" event={"ID":"6f6519d4-abe0-4ce9-bc38-38993a7ad791","Type":"ContainerStarted","Data":"3a535aca9219d6fb0f7c81ea0e12eb91ea9581a7f812b0d2b43f5164c21c7107"} Nov 02 01:42:39.816357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:39.816297 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podStartSLOduration=166.816279723 podStartE2EDuration="2m46.816279723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:39.811367384 +0000 UTC m=+1659.917156304" watchObservedRunningTime="2026-11-02 01:42:39.816279723 +0000 UTC m=+1659.922068638" Nov 02 01:42:40.075321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:40.075236 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae" path="/var/lib/kubelet/pods/24431e6d-f5f5-4e3e-a4b2-149ea7f6d6ae/volumes" Nov 02 01:42:40.255193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:40.252789 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:40.269145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:40.268277 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:42:40.269145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:40.268343 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 01:42:41.253153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:41.250176 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:42:41.253153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:41.250255 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 01:42:42.251748 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:42.251711 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:42:42.287146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:42.284136 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 01:42:43.254696 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:43.254660 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:42:43.255001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:43.254970 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 01:42:44.192907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:44.192847 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s"] Nov 02 01:42:44.206260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:44.206223 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87"] Nov 02 01:42:44.206754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:44.206719 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" containerID="cri-o://ad33bbef03fbce5880ef4d9c62fc1606e9b990fbf765f8898d1df4d7e680d155" gracePeriod=30 Nov 02 01:42:44.480027 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:44.207499 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" containerID="cri-o://7104ace13d72dc228892c6ee6b4dc24a3b13b26be5d68204ce0368bc989d7e75" gracePeriod=30 Nov 02 01:42:44.830581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:44.249512 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:42:44.969415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:44.969369 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 01:42:45.269449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:45.269413 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:42:45.270309 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:45.270273 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 01:42:46.422906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:46.422817 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:46.443844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:46.443815 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:46.468921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:46.468890 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:42:50.052580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.052483 4552 generic.go:356] "Generic (PLEG): container finished" podID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerID="ad33bbef03fbce5880ef4d9c62fc1606e9b990fbf765f8898d1df4d7e680d155" exitCode=0 Nov 02 01:42:50.056382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.055602 4552 generic.go:356] "Generic (PLEG): container finished" podID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerID="7104ace13d72dc228892c6ee6b4dc24a3b13b26be5d68204ce0368bc989d7e75" exitCode=0 Nov 02 01:42:50.084036 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.083992 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" event={"ID":"03f84d5a-e446-4384-bbf9-b800146d4cb1","Type":"ContainerDied","Data":"ad33bbef03fbce5880ef4d9c62fc1606e9b990fbf765f8898d1df4d7e680d155"} Nov 02 01:42:50.084221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.084067 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" event={"ID":"e3bb7bfa-d628-46aa-9b3e-19679afd118a","Type":"ContainerDied","Data":"7104ace13d72dc228892c6ee6b4dc24a3b13b26be5d68204ce0368bc989d7e75"} Nov 02 01:42:50.144164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.142643 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:42:50.217176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.213308 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn"] Nov 02 01:42:50.217176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.214613 4552 cpu_manager.go:414] "containerMap: removing container" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" Nov 02 01:42:50.217176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.214637 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" Nov 02 01:42:50.217176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.214922 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" containerName="istio-proxy" Nov 02 01:42:50.219610 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.219574 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.222801 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.222779 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn"] Nov 02 01:42:50.223984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.223958 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e412d5-serving-cert\"" Nov 02 01:42:50.232583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232557 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.232717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232625 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.232717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232680 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.232717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232706 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.232887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232749 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.232887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232794 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.232887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232861 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.233029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.232890 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.233081 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.233036 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.233389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.233152 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs\") pod \"03f84d5a-e446-4384-bbf9-b800146d4cb1\" (UID: \"03f84d5a-e446-4384-bbf9-b800146d4cb1\") " Nov 02 01:42:50.233950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.233783 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:50.234427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.234091 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.234427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.234365 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.234427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.234381 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f84d5a-e446-4384-bbf9-b800146d4cb1-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.234427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.234398 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.234727 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.234691 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.235597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.235575 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.235921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.235882 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "kube-api-access-tbxhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:50.236258 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.236225 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:50.236549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.236519 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:42:50.237890 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.237865 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy" pod "03f84d5a-e446-4384-bbf9-b800146d4cb1" (UID: "03f84d5a-e446-4384-bbf9-b800146d4cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.259413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.259027 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:42:50.298701 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.298663 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5"] Nov 02 01:42:50.299613 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.299590 4552 cpu_manager.go:414] "containerMap: removing container" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" Nov 02 01:42:50.299613 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.299609 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" Nov 02 01:42:50.299792 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.299731 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" containerName="istio-proxy" Nov 02 01:42:50.304538 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.304465 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.321699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.321659 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5"] Nov 02 01:42:50.335723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.335690 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.335924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.335787 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.335924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.335840 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.335924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.335882 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.336089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.335963 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.336089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336060 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.336204 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336092 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.336204 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336194 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.336306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336264 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.336306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336294 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.336425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336335 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:50.336425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336346 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.336425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336386 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy\") pod \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\" (UID: \"e3bb7bfa-d628-46aa-9b3e-19679afd118a\") " Nov 02 01:42:50.336557 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336480 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.336787 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.336767 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sclr\" (UniqueName: \"kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.337284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337218 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.337445 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337370 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337614 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337786 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337808 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337824 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337850 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337866 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337878 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f84d5a-e446-4384-bbf9-b800146d4cb1-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337892 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337907 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxhd\" (UniqueName: \"kubernetes.io/projected/03f84d5a-e446-4384-bbf9-b800146d4cb1-kube-api-access-tbxhd\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337921 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337935 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.338040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.337949 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.339349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.339315 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "kube-api-access-26dqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:50.339500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.339469 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:50.339594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.339507 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:42:50.339873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.339849 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token" pod "e3bb7bfa-d628-46aa-9b3e-19679afd118a" (UID: "e3bb7bfa-d628-46aa-9b3e-19679afd118a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:50.439475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439433 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.439672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439502 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.439672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439618 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.439672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439657 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.439847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439690 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.439847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439825 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.439954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439886 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.440011 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.439954 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sclr\" (UniqueName: \"kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.440105 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440063 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64wzr\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.440201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440105 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.440277 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440257 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.440382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440358 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.440492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440467 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.440687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440672 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.440763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440692 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.440763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440709 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.440763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.440726 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dqz\" (UniqueName: \"kubernetes.io/projected/e3bb7bfa-d628-46aa-9b3e-19679afd118a-kube-api-access-26dqz\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:50.441464 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.441441 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.442793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.442772 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.443162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.443139 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.452826 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.452792 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sclr\" (UniqueName: \"kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr\") pod \"maas-api-e2e-inf-b-e412d5-67477d9746-h75xn\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.542635 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.542596 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64wzr\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.542635 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.542642 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.542994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.542721 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.542994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.542901 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.542994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.542947 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.542994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.542981 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.543238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.543006 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.543238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.543038 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.543238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.543133 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.543625 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.543588 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.543820 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.543640 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.543966 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.543826 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.544177 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.544152 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.545002 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.544952 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.546298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.546270 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.546391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.546375 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.555021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.554954 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.555316 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.555289 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64wzr\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.575537 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.575500 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:50.629249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.629204 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:50.929900 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:42:50.929852 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fc09d5_d280_43af_b1aa_a09cdda17540.slice/crio-3b3208d2e1cfa1d6568f76b1e4c82e17df4a9648648067636a89ad302f061b90 WatchSource:0}: Error finding container 3b3208d2e1cfa1d6568f76b1e4c82e17df4a9648648067636a89ad302f061b90: Status 404 returned error can't find the container with id 3b3208d2e1cfa1d6568f76b1e4c82e17df4a9648648067636a89ad302f061b90 Nov 02 01:42:50.931285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.931245 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn"] Nov 02 01:42:50.931906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:50.931887 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:42:51.045302 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.041304 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5"] Nov 02 01:42:51.047878 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:42:51.046675 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe252814_5eec_4e00_9e30_013c83dbd6d0.slice/crio-fd3a0532c6f5d009e254923afd2da44624289569738d4eb0af46ed056ba7ba2c WatchSource:0}: Error finding container fd3a0532c6f5d009e254923afd2da44624289569738d4eb0af46ed056ba7ba2c: Status 404 returned error can't find the container with id fd3a0532c6f5d009e254923afd2da44624289569738d4eb0af46ed056ba7ba2c Nov 02 01:42:51.055510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.055462 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:42:51.055878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.055561 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:42:51.055878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.055604 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:42:51.070436 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.070354 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" Nov 02 01:42:51.070436 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.070396 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87" event={"ID":"03f84d5a-e446-4384-bbf9-b800146d4cb1","Type":"ContainerDied","Data":"1ea9bb3035e05319caef4c5960dd13d08804910937d629e6338aec22aaadfebc"} Nov 02 01:42:51.070643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.070466 4552 scope.go:122] "RemoveContainer" containerID="ad33bbef03fbce5880ef4d9c62fc1606e9b990fbf765f8898d1df4d7e680d155" Nov 02 01:42:51.073972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.073598 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" event={"ID":"fe252814-5eec-4e00-9e30-013c83dbd6d0","Type":"ContainerStarted","Data":"fd3a0532c6f5d009e254923afd2da44624289569738d4eb0af46ed056ba7ba2c"} Nov 02 01:42:51.077434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.077403 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" event={"ID":"59fc09d5-d280-43af-b1aa-a09cdda17540","Type":"ContainerStarted","Data":"3b3208d2e1cfa1d6568f76b1e4c82e17df4a9648648067636a89ad302f061b90"} Nov 02 01:42:51.079463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.079437 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" event={"ID":"e3bb7bfa-d628-46aa-9b3e-19679afd118a","Type":"ContainerDied","Data":"05d8a8a847e53561aba8ffdd112ddb60d8a5cdc1cb6a18fefa5b3e47f0b47188"} Nov 02 01:42:51.079583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.079483 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s" Nov 02 01:42:51.087690 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.087668 4552 scope.go:122] "RemoveContainer" containerID="7104ace13d72dc228892c6ee6b4dc24a3b13b26be5d68204ce0368bc989d7e75" Nov 02 01:42:51.105107 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.105067 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87"] Nov 02 01:42:51.111310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.111278 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-brt87"] Nov 02 01:42:51.162195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.157270 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s"] Nov 02 01:42:51.164029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:51.163994 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-qqk9s"] Nov 02 01:42:52.053657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.053618 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f84d5a-e446-4384-bbf9-b800146d4cb1" path="/var/lib/kubelet/pods/03f84d5a-e446-4384-bbf9-b800146d4cb1/volumes" Nov 02 01:42:52.054262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.054242 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bb7bfa-d628-46aa-9b3e-19679afd118a" path="/var/lib/kubelet/pods/e3bb7bfa-d628-46aa-9b3e-19679afd118a/volumes" Nov 02 01:42:52.122219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.121846 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" event={"ID":"fe252814-5eec-4e00-9e30-013c83dbd6d0","Type":"ContainerStarted","Data":"ad63ca75c4ed7106d03ed3e847e0276da8797b198c2ee936ce59167688163a87"} Nov 02 01:42:52.123822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.123799 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" event={"ID":"59fc09d5-d280-43af-b1aa-a09cdda17540","Type":"ContainerStarted","Data":"ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934"} Nov 02 01:42:52.123940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.123924 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:52.182262 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.182175 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podStartSLOduration=164.182097013 podStartE2EDuration="2m44.182097013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:52.147782534 +0000 UTC m=+1672.253571448" watchObservedRunningTime="2026-11-02 01:42:52.182097013 +0000 UTC m=+1672.287885936" Nov 02 01:42:52.193413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.193354 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" podStartSLOduration=176.6254956 podStartE2EDuration="2m57.193337852s" totalImagesPullingTime="567.842252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:52.185139033 +0000 UTC m=+1672.290927977" watchObservedRunningTime="2026-11-02 01:42:52.193337852 +0000 UTC m=+1672.299126795" Nov 02 01:42:52.631024 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.630995 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:52.632407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.632375 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 01:42:52.632543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:52.632421 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 01:42:53.630872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:53.630838 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 01:42:53.631291 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:53.630904 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 01:42:54.631271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:54.631227 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 01:42:54.631683 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:54.631283 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 01:42:55.630967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:55.630926 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 01:42:55.631189 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:55.631000 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 01:42:56.587580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:56.587540 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6"] Nov 02 01:42:56.631326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:56.631294 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 01:42:56.631580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:56.631550 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 01:42:57.036241 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:57.036209 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82"] Nov 02 01:42:57.631322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:57.631285 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 01:42:57.631747 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:57.631341 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 01:42:57.767065 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:42:57.767017 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:42:58.142951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:58.142909 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:42:58.637331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:58.637271 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:58.637936 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:58.637917 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:58.638985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:58.638963 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:42:59.033092 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:42:59.033051 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg"] Nov 02 01:43:11.176739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:11.176695 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6"] Nov 02 01:43:11.622434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:11.622387 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" containerID="cri-o://043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe" gracePeriod=45 Nov 02 01:43:11.936893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:11.936867 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:43:12.003062 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.003025 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t\" (UniqueName: \"kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t\") pod \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " Nov 02 01:43:12.003261 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.003179 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home\") pod \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " Nov 02 01:43:12.003435 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.003274 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir\") pod \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " Nov 02 01:43:12.003435 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.003393 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs\") pod \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " Nov 02 01:43:12.003581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.003570 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm\") pod \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " Nov 02 01:43:12.003821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.003792 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache\") pod \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\" (UID: \"9a1ffeff-fbf6-4319-b0ae-2f934090af33\") " Nov 02 01:43:12.004564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.004277 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache" pod "9a1ffeff-fbf6-4319-b0ae-2f934090af33" (UID: "9a1ffeff-fbf6-4319-b0ae-2f934090af33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.004564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.004483 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir" pod "9a1ffeff-fbf6-4319-b0ae-2f934090af33" (UID: "9a1ffeff-fbf6-4319-b0ae-2f934090af33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.004564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.004538 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home" pod "9a1ffeff-fbf6-4319-b0ae-2f934090af33" (UID: "9a1ffeff-fbf6-4319-b0ae-2f934090af33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.005205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.005179 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.005299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.005212 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.005299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.005228 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.006467 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.006442 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t" pod "9a1ffeff-fbf6-4319-b0ae-2f934090af33" (UID: "9a1ffeff-fbf6-4319-b0ae-2f934090af33"). InnerVolumeSpecName "kube-api-access-wk58t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:12.027507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.027321 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs" pod "9a1ffeff-fbf6-4319-b0ae-2f934090af33" (UID: "9a1ffeff-fbf6-4319-b0ae-2f934090af33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:12.027744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.027711 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm" pod "9a1ffeff-fbf6-4319-b0ae-2f934090af33" (UID: "9a1ffeff-fbf6-4319-b0ae-2f934090af33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.123393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.106841 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1ffeff-fbf6-4319-b0ae-2f934090af33-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.123393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.106873 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk58t\" (UniqueName: \"kubernetes.io/projected/9a1ffeff-fbf6-4319-b0ae-2f934090af33-kube-api-access-wk58t\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.123393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.106882 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1ffeff-fbf6-4319-b0ae-2f934090af33-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.393667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.393616 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" containerID="cri-o://4a04c2f562a5c375aa959283dd0fccb93de5f575335b54cf4e3072afe79d79e8" gracePeriod=45 Nov 02 01:43:12.410744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.410706 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.128:8000/ready\": dial tcp 10.134.0.128:8000: connect: connection refused" Nov 02 01:43:12.558913 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.558879 4552 generic.go:356] "Generic (PLEG): container finished" podID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerID="4a04c2f562a5c375aa959283dd0fccb93de5f575335b54cf4e3072afe79d79e8" exitCode=0 Nov 02 01:43:12.559184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.558934 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" event={"ID":"eb503526-1cc8-4ba0-bbf2-701c6f37a0db","Type":"ContainerDied","Data":"4a04c2f562a5c375aa959283dd0fccb93de5f575335b54cf4e3072afe79d79e8"} Nov 02 01:43:12.560993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.560960 4552 generic.go:356] "Generic (PLEG): container finished" podID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerID="043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe" exitCode=0 Nov 02 01:43:12.561171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.561017 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" Nov 02 01:43:12.561171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.561032 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" event={"ID":"9a1ffeff-fbf6-4319-b0ae-2f934090af33","Type":"ContainerDied","Data":"043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe"} Nov 02 01:43:12.561171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.561069 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6" event={"ID":"9a1ffeff-fbf6-4319-b0ae-2f934090af33","Type":"ContainerDied","Data":"71cf2bb1b0d3803cace6f35e50371c0cfa37997dd526b3a481808d14878a42d6"} Nov 02 01:43:12.561171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.561091 4552 scope.go:122] "RemoveContainer" containerID="043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe" Nov 02 01:43:12.575807 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.575772 4552 scope.go:122] "RemoveContainer" containerID="043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe" Nov 02 01:43:12.576183 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:12.576160 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe\": container with ID starting with 043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe not found: ID does not exist" containerID="043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe" Nov 02 01:43:12.576313 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.576189 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe"} err="failed to get container status \"043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe\": rpc error: code = NotFound desc = could not find container \"043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe\": container with ID starting with 043a54d032ecdef7649de96cd7874e2789a92250c73e671960b7dcb3afa9e6fe not found: ID does not exist" Nov 02 01:43:12.587657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.587621 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6"] Nov 02 01:43:12.613074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.613023 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ffe5c1/test-model-ffe5c1-kserve-697494d8c-h5ld6"] Nov 02 01:43:12.721128 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.720973 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:43:12.820579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.820546 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5\" (UniqueName: \"kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5\") pod \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " Nov 02 01:43:12.820782 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.820621 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir\") pod \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " Nov 02 01:43:12.820782 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.820737 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home\") pod \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " Nov 02 01:43:12.820900 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.820865 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache\") pod \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " Nov 02 01:43:12.820951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.820901 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm\") pod \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " Nov 02 01:43:12.821095 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.821078 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs\") pod \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\" (UID: \"eb503526-1cc8-4ba0-bbf2-701c6f37a0db\") " Nov 02 01:43:12.821229 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.820957 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir" pod "eb503526-1cc8-4ba0-bbf2-701c6f37a0db" (UID: "eb503526-1cc8-4ba0-bbf2-701c6f37a0db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.821514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.821176 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache" pod "eb503526-1cc8-4ba0-bbf2-701c6f37a0db" (UID: "eb503526-1cc8-4ba0-bbf2-701c6f37a0db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.822417 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.822391 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home" pod "eb503526-1cc8-4ba0-bbf2-701c6f37a0db" (UID: "eb503526-1cc8-4ba0-bbf2-701c6f37a0db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.822512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.822462 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.822512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.822481 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.823461 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.823427 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm" pod "eb503526-1cc8-4ba0-bbf2-701c6f37a0db" (UID: "eb503526-1cc8-4ba0-bbf2-701c6f37a0db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:12.823533 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.823500 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs" pod "eb503526-1cc8-4ba0-bbf2-701c6f37a0db" (UID: "eb503526-1cc8-4ba0-bbf2-701c6f37a0db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:12.823597 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.823584 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5" pod "eb503526-1cc8-4ba0-bbf2-701c6f37a0db" (UID: "eb503526-1cc8-4ba0-bbf2-701c6f37a0db"). InnerVolumeSpecName "kube-api-access-7t8l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:12.923979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.923892 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t8l5\" (UniqueName: \"kubernetes.io/projected/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-kube-api-access-7t8l5\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.923979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.923923 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.923979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.923938 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.923979 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:12.923947 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb503526-1cc8-4ba0-bbf2-701c6f37a0db-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:13.572698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:13.572661 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" Nov 02 01:43:13.572698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:13.572679 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82" event={"ID":"eb503526-1cc8-4ba0-bbf2-701c6f37a0db","Type":"ContainerDied","Data":"800bf6ed0d4a0f83a0ebd84883924b3e9a71d4c7f36414d6905674dad2aa4925"} Nov 02 01:43:13.573219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:13.572735 4552 scope.go:122] "RemoveContainer" containerID="4a04c2f562a5c375aa959283dd0fccb93de5f575335b54cf4e3072afe79d79e8" Nov 02 01:43:13.601381 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:13.601348 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82"] Nov 02 01:43:13.608139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:13.608075 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b34fc0/test-model-b34fc0-kserve-fb97777c9-qnj82"] Nov 02 01:43:14.048451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.048410 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" path="/var/lib/kubelet/pods/9a1ffeff-fbf6-4319-b0ae-2f934090af33/volumes" Nov 02 01:43:14.049082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.049064 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" path="/var/lib/kubelet/pods/eb503526-1cc8-4ba0-bbf2-701c6f37a0db/volumes" Nov 02 01:43:14.063186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.063141 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" containerID="cri-o://f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda" gracePeriod=45 Nov 02 01:43:14.472244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.472216 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:43:14.578969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.578927 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm\") pod \"01903806-7eda-48ca-8097-c71608b6920f\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " Nov 02 01:43:14.580020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.579051 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs\" (UniqueName: \"kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs\") pod \"01903806-7eda-48ca-8097-c71608b6920f\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " Nov 02 01:43:14.580020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.579344 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home\") pod \"01903806-7eda-48ca-8097-c71608b6920f\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " Nov 02 01:43:14.580020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.579385 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv\" (UniqueName: \"kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv\") pod \"01903806-7eda-48ca-8097-c71608b6920f\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " Nov 02 01:43:14.580020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.579423 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache\") pod \"01903806-7eda-48ca-8097-c71608b6920f\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " Nov 02 01:43:14.580020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.579504 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir\") pod \"01903806-7eda-48ca-8097-c71608b6920f\" (UID: \"01903806-7eda-48ca-8097-c71608b6920f\") " Nov 02 01:43:14.580431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.580286 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home" pod "01903806-7eda-48ca-8097-c71608b6920f" (UID: "01903806-7eda-48ca-8097-c71608b6920f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:14.580662 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.580614 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir" pod "01903806-7eda-48ca-8097-c71608b6920f" (UID: "01903806-7eda-48ca-8097-c71608b6920f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:14.580830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.580760 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache" pod "01903806-7eda-48ca-8097-c71608b6920f" (UID: "01903806-7eda-48ca-8097-c71608b6920f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:14.584753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.584722 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm" pod "01903806-7eda-48ca-8097-c71608b6920f" (UID: "01903806-7eda-48ca-8097-c71608b6920f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:14.584753 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.584731 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs" pod "01903806-7eda-48ca-8097-c71608b6920f" (UID: "01903806-7eda-48ca-8097-c71608b6920f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:14.584970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.584796 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv" pod "01903806-7eda-48ca-8097-c71608b6920f" (UID: "01903806-7eda-48ca-8097-c71608b6920f"). InnerVolumeSpecName "kube-api-access-gp6nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:14.592568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.592528 4552 generic.go:356] "Generic (PLEG): container finished" podID="01903806-7eda-48ca-8097-c71608b6920f" containerID="f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda" exitCode=0 Nov 02 01:43:14.592757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.592732 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" Nov 02 01:43:14.594194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.594090 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" event={"ID":"01903806-7eda-48ca-8097-c71608b6920f","Type":"ContainerDied","Data":"f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda"} Nov 02 01:43:14.594338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.594203 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg" event={"ID":"01903806-7eda-48ca-8097-c71608b6920f","Type":"ContainerDied","Data":"8153334f61c24940d8ebcce3838d5e660d5de43ecf3e1fc1d7b1e4ac62496130"} Nov 02 01:43:14.594338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.594228 4552 scope.go:122] "RemoveContainer" containerID="f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda" Nov 02 01:43:14.637261 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.637229 4552 scope.go:122] "RemoveContainer" containerID="f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda" Nov 02 01:43:14.638007 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:14.637979 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda\": container with ID starting with f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda not found: ID does not exist" containerID="f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda" Nov 02 01:43:14.638154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.638019 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda"} err="failed to get container status \"f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda\": rpc error: code = NotFound desc = could not find container \"f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda\": container with ID starting with f728006f87e5f78666230250cc02961a02e84bebca8185e22b7d1eef65934bda not found: ID does not exist" Nov 02 01:43:14.648167 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.648134 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg"] Nov 02 01:43:14.656587 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.656554 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2589b2/test-model-2589b2-kserve-7974f8c6c-p7svg"] Nov 02 01:43:14.682168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.682044 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:14.682168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.682071 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01903806-7eda-48ca-8097-c71608b6920f-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:14.682168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.682080 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:14.682168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.682089 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6nv\" (UniqueName: \"kubernetes.io/projected/01903806-7eda-48ca-8097-c71608b6920f-kube-api-access-gp6nv\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:14.682168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.682099 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:14.682168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:14.682109 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01903806-7eda-48ca-8097-c71608b6920f-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:16.132139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:16.128826 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01903806-7eda-48ca-8097-c71608b6920f" path="/var/lib/kubelet/pods/01903806-7eda-48ca-8097-c71608b6920f/volumes" Nov 02 01:43:17.845649 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:17.845197 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn"] Nov 02 01:43:17.845649 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:17.845447 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" podUID="59fc09d5-d280-43af-b1aa-a09cdda17540" containerName="maas-api" containerID="cri-o://ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934" gracePeriod=30 Nov 02 01:43:18.165162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.138607 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" podUID="59fc09d5-d280-43af-b1aa-a09cdda17540" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.141:8443/health\": dial tcp 10.134.0.141:8443: connect: connection refused" Nov 02 01:43:18.549133 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.549099 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:43:18.628672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.628638 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca\") pod \"59fc09d5-d280-43af-b1aa-a09cdda17540\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " Nov 02 01:43:18.628885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.628834 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls\") pod \"59fc09d5-d280-43af-b1aa-a09cdda17540\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " Nov 02 01:43:18.629052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.629010 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr\" (UniqueName: \"kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr\") pod \"59fc09d5-d280-43af-b1aa-a09cdda17540\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " Nov 02 01:43:18.629143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.629086 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs\") pod \"59fc09d5-d280-43af-b1aa-a09cdda17540\" (UID: \"59fc09d5-d280-43af-b1aa-a09cdda17540\") " Nov 02 01:43:18.629677 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.629493 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca" pod "59fc09d5-d280-43af-b1aa-a09cdda17540" (UID: "59fc09d5-d280-43af-b1aa-a09cdda17540"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:18.630033 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.630007 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fc09d5-d280-43af-b1aa-a09cdda17540-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:18.632414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.632382 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs" pod "59fc09d5-d280-43af-b1aa-a09cdda17540" (UID: "59fc09d5-d280-43af-b1aa-a09cdda17540"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:18.640342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.635471 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls" pod "59fc09d5-d280-43af-b1aa-a09cdda17540" (UID: "59fc09d5-d280-43af-b1aa-a09cdda17540"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:18.640553 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.640415 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr" pod "59fc09d5-d280-43af-b1aa-a09cdda17540" (UID: "59fc09d5-d280-43af-b1aa-a09cdda17540"). InnerVolumeSpecName "kube-api-access-5sclr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:18.657080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.656956 4552 generic.go:356] "Generic (PLEG): container finished" podID="59fc09d5-d280-43af-b1aa-a09cdda17540" containerID="ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934" exitCode=0 Nov 02 01:43:18.657080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.657056 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" event={"ID":"59fc09d5-d280-43af-b1aa-a09cdda17540","Type":"ContainerDied","Data":"ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934"} Nov 02 01:43:18.657310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.657145 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" event={"ID":"59fc09d5-d280-43af-b1aa-a09cdda17540","Type":"ContainerDied","Data":"3b3208d2e1cfa1d6568f76b1e4c82e17df4a9648648067636a89ad302f061b90"} Nov 02 01:43:18.657310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.657182 4552 scope.go:122] "RemoveContainer" containerID="ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934" Nov 02 01:43:18.657432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.657350 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn" Nov 02 01:43:18.693185 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.693150 4552 scope.go:122] "RemoveContainer" containerID="ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934" Nov 02 01:43:18.693562 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:18.693538 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934\": container with ID starting with ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934 not found: ID does not exist" containerID="ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934" Nov 02 01:43:18.693648 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.693576 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934"} err="failed to get container status \"ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934\": rpc error: code = NotFound desc = could not find container \"ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934\": container with ID starting with ef2c33dc22398577a57c20d67a8d29a55ab5e4243850e0b5b75758a21662a934 not found: ID does not exist" Nov 02 01:43:18.711356 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.705073 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn"] Nov 02 01:43:18.714209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.714176 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e412d5-67477d9746-h75xn"] Nov 02 01:43:18.736575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.736539 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:18.736857 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.736599 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sclr\" (UniqueName: \"kubernetes.io/projected/59fc09d5-d280-43af-b1aa-a09cdda17540-kube-api-access-5sclr\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:18.736857 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:18.736619 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fc09d5-d280-43af-b1aa-a09cdda17540-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:20.051202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:20.051159 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fc09d5-d280-43af-b1aa-a09cdda17540" path="/var/lib/kubelet/pods/59fc09d5-d280-43af-b1aa-a09cdda17540/volumes" Nov 02 01:43:21.180706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:21.179963 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b"] Nov 02 01:43:21.182484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:21.181542 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" podUID="bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" containerName="maas-api" containerID="cri-o://adf10848209418d4380ee774f22066c71e25baeae365b1e6f8ed55bac8ab0b91" gracePeriod=30 Nov 02 01:43:22.820846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.820808 4552 generic.go:356] "Generic (PLEG): container finished" podID="bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" containerID="adf10848209418d4380ee774f22066c71e25baeae365b1e6f8ed55bac8ab0b91" exitCode=0 Nov 02 01:43:22.821281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.820884 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" event={"ID":"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8","Type":"ContainerDied","Data":"adf10848209418d4380ee774f22066c71e25baeae365b1e6f8ed55bac8ab0b91"} Nov 02 01:43:22.821281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.820948 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" event={"ID":"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8","Type":"ContainerDied","Data":"40eed1681327aa099bd4f704f064e5deae28fb41e08de8936d1852b846075c40"} Nov 02 01:43:22.821281 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.820974 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40eed1681327aa099bd4f704f064e5deae28fb41e08de8936d1852b846075c40" Nov 02 01:43:22.833193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.833167 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:43:22.929564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.929532 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs\") pod \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " Nov 02 01:43:22.929784 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.929724 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls\") pod \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " Nov 02 01:43:22.929985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.929964 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca\") pod \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " Nov 02 01:43:22.930217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.930192 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7\" (UniqueName: \"kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7\") pod \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\" (UID: \"bb4cdeea-aeaa-4a74-abee-8d3925ea93c8\") " Nov 02 01:43:22.930382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.930353 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca" pod "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" (UID: "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:22.931009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.930984 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:22.935299 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.935274 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs" pod "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" (UID: "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:22.935415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.935321 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls" pod "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" (UID: "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:22.935482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:22.935409 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7" pod "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" (UID: "bb4cdeea-aeaa-4a74-abee-8d3925ea93c8"). InnerVolumeSpecName "kube-api-access-5v4z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:23.032928 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:23.032814 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v4z7\" (UniqueName: \"kubernetes.io/projected/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-kube-api-access-5v4z7\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:23.032928 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:23.032859 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:23.032928 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:23.032873 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:23.830155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:23.830107 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b" Nov 02 01:43:23.880195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:23.880158 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b"] Nov 02 01:43:23.892149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:23.892095 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-addad9-7b59dff9c7-fkv2b"] Nov 02 01:43:24.048910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:24.048873 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" path="/var/lib/kubelet/pods/bb4cdeea-aeaa-4a74-abee-8d3925ea93c8/volumes" Nov 02 01:43:26.201108 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.201062 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" containerID="cri-o://7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b" gracePeriod=45 Nov 02 01:43:26.527537 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.527510 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:43:26.693139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.692256 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home\") pod \"06e19217-f9be-4f7f-b790-605fe7baba8d\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " Nov 02 01:43:26.693139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.692421 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache\") pod \"06e19217-f9be-4f7f-b790-605fe7baba8d\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " Nov 02 01:43:26.693139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.692516 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs\") pod \"06e19217-f9be-4f7f-b790-605fe7baba8d\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " Nov 02 01:43:26.693139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.692588 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp\" (UniqueName: \"kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp\") pod \"06e19217-f9be-4f7f-b790-605fe7baba8d\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " Nov 02 01:43:26.693139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.692623 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm\") pod \"06e19217-f9be-4f7f-b790-605fe7baba8d\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " Nov 02 01:43:26.693139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.692916 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir\") pod \"06e19217-f9be-4f7f-b790-605fe7baba8d\" (UID: \"06e19217-f9be-4f7f-b790-605fe7baba8d\") " Nov 02 01:43:26.786144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.784629 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs" pod "06e19217-f9be-4f7f-b790-605fe7baba8d" (UID: "06e19217-f9be-4f7f-b790-605fe7baba8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:26.786144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.784874 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache" pod "06e19217-f9be-4f7f-b790-605fe7baba8d" (UID: "06e19217-f9be-4f7f-b790-605fe7baba8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:26.788292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.788229 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home" pod "06e19217-f9be-4f7f-b790-605fe7baba8d" (UID: "06e19217-f9be-4f7f-b790-605fe7baba8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:26.788546 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.788505 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir" pod "06e19217-f9be-4f7f-b790-605fe7baba8d" (UID: "06e19217-f9be-4f7f-b790-605fe7baba8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:26.810151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.806023 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:26.810151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.806057 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:26.810151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.806070 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:26.810151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.806084 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e19217-f9be-4f7f-b790-605fe7baba8d-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:26.838724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.824247 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp" pod "06e19217-f9be-4f7f-b790-605fe7baba8d" (UID: "06e19217-f9be-4f7f-b790-605fe7baba8d"). InnerVolumeSpecName "kube-api-access-7w4zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:26.838724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.824897 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm" pod "06e19217-f9be-4f7f-b790-605fe7baba8d" (UID: "06e19217-f9be-4f7f-b790-605fe7baba8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:26.910141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.910014 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4zp\" (UniqueName: \"kubernetes.io/projected/06e19217-f9be-4f7f-b790-605fe7baba8d-kube-api-access-7w4zp\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:26.910141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.910047 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e19217-f9be-4f7f-b790-605fe7baba8d-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:26.940964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.940886 4552 generic.go:356] "Generic (PLEG): container finished" podID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerID="7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b" exitCode=0 Nov 02 01:43:26.941221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.940998 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" event={"ID":"06e19217-f9be-4f7f-b790-605fe7baba8d","Type":"ContainerDied","Data":"7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b"} Nov 02 01:43:26.941221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.941035 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" event={"ID":"06e19217-f9be-4f7f-b790-605fe7baba8d","Type":"ContainerDied","Data":"b2680f27e56c353c5759ab1ac744a4e20c9a042472f9f1bb5691249970c55ea3"} Nov 02 01:43:26.941221 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.941059 4552 scope.go:122] "RemoveContainer" containerID="7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b" Nov 02 01:43:26.942324 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.941417 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" Nov 02 01:43:26.956801 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.956780 4552 scope.go:122] "RemoveContainer" containerID="7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b" Nov 02 01:43:26.957137 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:26.957100 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b\": container with ID starting with 7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b not found: ID does not exist" containerID="7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b" Nov 02 01:43:26.957203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.957145 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b"} err="failed to get container status \"7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b\": rpc error: code = NotFound desc = could not find container \"7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b\": container with ID starting with 7cf9cdfef928b2b1a36b9d70ac65df48576b5b827e8a39b7cc4542fb54b61a1b not found: ID does not exist" Nov 02 01:43:26.974413 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.974332 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6"] Nov 02 01:43:26.980170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:26.979792 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6"] Nov 02 01:43:27.473307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:27.473260 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9a4595/test-model-9a4595-kserve-68d6f4986f-2fbn6" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": context deadline exceeded" Nov 02 01:43:27.758459 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:27.758418 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:43:28.050982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:28.050895 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" path="/var/lib/kubelet/pods/06e19217-f9be-4f7f-b790-605fe7baba8d/volumes" Nov 02 01:43:28.822284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:28.822246 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb"] Nov 02 01:43:28.823111 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:28.823057 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" podUID="fd7dce9b-fb14-4c23-8960-11ea4bcd109a" containerName="maas-api" containerID="cri-o://0d3c4d4a4025cdbbd36241b2ff231786d472eb893f0031e02d8f5767a70be19a" gracePeriod=30 Nov 02 01:43:28.966747 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:28.966705 4552 generic.go:356] "Generic (PLEG): container finished" podID="fd7dce9b-fb14-4c23-8960-11ea4bcd109a" containerID="0d3c4d4a4025cdbbd36241b2ff231786d472eb893f0031e02d8f5767a70be19a" exitCode=0 Nov 02 01:43:28.966948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:28.966760 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" event={"ID":"fd7dce9b-fb14-4c23-8960-11ea4bcd109a","Type":"ContainerDied","Data":"0d3c4d4a4025cdbbd36241b2ff231786d472eb893f0031e02d8f5767a70be19a"} Nov 02 01:43:29.475169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.475094 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:43:29.479818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.479786 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs\") pod \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " Nov 02 01:43:29.480004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.479871 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls\") pod \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " Nov 02 01:43:29.482227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.482195 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls" pod "fd7dce9b-fb14-4c23-8960-11ea4bcd109a" (UID: "fd7dce9b-fb14-4c23-8960-11ea4bcd109a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:29.482227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.482211 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs" pod "fd7dce9b-fb14-4c23-8960-11ea4bcd109a" (UID: "fd7dce9b-fb14-4c23-8960-11ea4bcd109a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:29.581763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.581725 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca\") pod \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " Nov 02 01:43:29.581965 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.581825 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57\" (UniqueName: \"kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57\") pod \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\" (UID: \"fd7dce9b-fb14-4c23-8960-11ea4bcd109a\") " Nov 02 01:43:29.582226 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.582183 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca" pod "fd7dce9b-fb14-4c23-8960-11ea4bcd109a" (UID: "fd7dce9b-fb14-4c23-8960-11ea4bcd109a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:29.582674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.582648 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:29.582791 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.582681 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:29.582791 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.582697 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:29.584266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.584240 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57" pod "fd7dce9b-fb14-4c23-8960-11ea4bcd109a" (UID: "fd7dce9b-fb14-4c23-8960-11ea4bcd109a"). InnerVolumeSpecName "kube-api-access-6fg57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:29.685515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.685390 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fg57\" (UniqueName: \"kubernetes.io/projected/fd7dce9b-fb14-4c23-8960-11ea4bcd109a-kube-api-access-6fg57\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:29.983434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.983383 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" event={"ID":"fd7dce9b-fb14-4c23-8960-11ea4bcd109a","Type":"ContainerDied","Data":"85c03d3ddf1087f5bdcd4d674edcd1070b84fe8135fdba75536324340ef4c80a"} Nov 02 01:43:29.983911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.983502 4552 scope.go:122] "RemoveContainer" containerID="0d3c4d4a4025cdbbd36241b2ff231786d472eb893f0031e02d8f5767a70be19a" Nov 02 01:43:29.983911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:29.983657 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb" Nov 02 01:43:30.074520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:30.072852 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb"] Nov 02 01:43:30.079699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:30.079666 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0bbeda-7b78bbb564-8mbqb"] Nov 02 01:43:32.054310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:32.054268 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7dce9b-fb14-4c23-8960-11ea4bcd109a" path="/var/lib/kubelet/pods/fd7dce9b-fb14-4c23-8960-11ea4bcd109a/volumes" Nov 02 01:43:35.263239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:35.263203 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr"] Nov 02 01:43:35.283183 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:35.281418 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4"] Nov 02 01:43:37.570766 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:37.570734 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl"] Nov 02 01:43:37.571434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:37.571388 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" podUID="946026f3-a365-45bd-9c13-98c484fd7080" containerName="maas-api" containerID="cri-o://110b25dbeb2983e5c2e09e0003b5bc4938a5032b6d7f1ce21f0a1819a5e4a435" gracePeriod=30 Nov 02 01:43:38.152578 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.149864 4552 generic.go:356] "Generic (PLEG): container finished" podID="946026f3-a365-45bd-9c13-98c484fd7080" containerID="110b25dbeb2983e5c2e09e0003b5bc4938a5032b6d7f1ce21f0a1819a5e4a435" exitCode=0 Nov 02 01:43:38.200499 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.193248 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" event={"ID":"946026f3-a365-45bd-9c13-98c484fd7080","Type":"ContainerDied","Data":"110b25dbeb2983e5c2e09e0003b5bc4938a5032b6d7f1ce21f0a1819a5e4a435"} Nov 02 01:43:38.204507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.204476 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:43:38.226772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.226737 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls\") pod \"946026f3-a365-45bd-9c13-98c484fd7080\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " Nov 02 01:43:38.227132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.227092 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf\" (UniqueName: \"kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf\") pod \"946026f3-a365-45bd-9c13-98c484fd7080\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " Nov 02 01:43:38.227266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.227182 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca\") pod \"946026f3-a365-45bd-9c13-98c484fd7080\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " Nov 02 01:43:38.227341 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.227302 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs\") pod \"946026f3-a365-45bd-9c13-98c484fd7080\" (UID: \"946026f3-a365-45bd-9c13-98c484fd7080\") " Nov 02 01:43:38.231311 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.231280 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls" pod "946026f3-a365-45bd-9c13-98c484fd7080" (UID: "946026f3-a365-45bd-9c13-98c484fd7080"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:38.232273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.232236 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca" pod "946026f3-a365-45bd-9c13-98c484fd7080" (UID: "946026f3-a365-45bd-9c13-98c484fd7080"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:38.235271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.235067 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs" pod "946026f3-a365-45bd-9c13-98c484fd7080" (UID: "946026f3-a365-45bd-9c13-98c484fd7080"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:38.244325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.241279 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf" pod "946026f3-a365-45bd-9c13-98c484fd7080" (UID: "946026f3-a365-45bd-9c13-98c484fd7080"). InnerVolumeSpecName "kube-api-access-sg4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:38.329743 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.329706 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:38.329743 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.329739 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4jf\" (UniqueName: \"kubernetes.io/projected/946026f3-a365-45bd-9c13-98c484fd7080-kube-api-access-sg4jf\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:38.329956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.329752 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946026f3-a365-45bd-9c13-98c484fd7080-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:38.329956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:38.329769 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/946026f3-a365-45bd-9c13-98c484fd7080-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:39.174154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.174094 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" Nov 02 01:43:39.174640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.174148 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl" event={"ID":"946026f3-a365-45bd-9c13-98c484fd7080","Type":"ContainerDied","Data":"ddea626a04badab9d64de3c96625155355f2c309644be29fa337ad76802d2002"} Nov 02 01:43:39.174640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.174228 4552 scope.go:122] "RemoveContainer" containerID="110b25dbeb2983e5c2e09e0003b5bc4938a5032b6d7f1ce21f0a1819a5e4a435" Nov 02 01:43:39.205334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.205296 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl"] Nov 02 01:43:39.216268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.214823 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-eeeb1a-cc67f795-9k6nl"] Nov 02 01:43:39.234962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.234931 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2"] Nov 02 01:43:39.235237 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.235192 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" podUID="ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" containerName="maas-api" containerID="cri-o://3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8" gracePeriod=30 Nov 02 01:43:39.547162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.547136 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:43:39.652715 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.652675 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n\" (UniqueName: \"kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n\") pod \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " Nov 02 01:43:39.652906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.652883 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca\") pod \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " Nov 02 01:43:39.653064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.653042 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls\") pod \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " Nov 02 01:43:39.653169 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.653154 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs\") pod \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\" (UID: \"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da\") " Nov 02 01:43:39.653303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.653272 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca" pod "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" (UID: "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:39.653870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.653783 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:39.655390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.655355 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls" pod "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" (UID: "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:39.655512 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.655452 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs" pod "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" (UID: "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:39.655575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.655528 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n" pod "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" (UID: "ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da"). InnerVolumeSpecName "kube-api-access-q8l8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:39.755527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.755488 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:39.755527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.755522 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:39.755527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:39.755533 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8l8n\" (UniqueName: \"kubernetes.io/projected/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da-kube-api-access-q8l8n\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:40.087723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.087637 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946026f3-a365-45bd-9c13-98c484fd7080" path="/var/lib/kubelet/pods/946026f3-a365-45bd-9c13-98c484fd7080/volumes" Nov 02 01:43:40.208802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.207560 4552 generic.go:356] "Generic (PLEG): container finished" podID="ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" containerID="3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8" exitCode=0 Nov 02 01:43:40.208802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.208002 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" event={"ID":"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da","Type":"ContainerDied","Data":"3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8"} Nov 02 01:43:40.208802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.208060 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" event={"ID":"ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da","Type":"ContainerDied","Data":"504c7f868132dabdb5ae39efd1ca2e0a852673b2c8b3d1e0c2618525377fea2a"} Nov 02 01:43:40.208802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.208090 4552 scope.go:122] "RemoveContainer" containerID="3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8" Nov 02 01:43:40.208802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.208264 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2" Nov 02 01:43:40.246001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.243760 4552 scope.go:122] "RemoveContainer" containerID="3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8" Nov 02 01:43:40.246227 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:40.246190 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8\": container with ID starting with 3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8 not found: ID does not exist" containerID="3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8" Nov 02 01:43:40.246373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.246242 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8"} err="failed to get container status \"3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8\": rpc error: code = NotFound desc = could not find container \"3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8\": container with ID starting with 3057ab19a38437663d98e84e2fc4f3b0cb8eb3510d895921ab6bddbfb8d120f8 not found: ID does not exist" Nov 02 01:43:40.246373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.246294 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2"] Nov 02 01:43:40.254205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.253890 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b34fc0-5f68f7bf8b-qmbt2"] Nov 02 01:43:40.264517 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.264454 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" podUID="d03985cd-4dd2-456e-9683-a4de3c0fa09c" containerName="payload-processing" containerID="cri-o://d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535" gracePeriod=25 Nov 02 01:43:40.282186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.282141 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" podUID="d72601e1-f811-4ae5-bb74-af8ced4a5363" containerName="payload-pre-processing" containerID="cri-o://0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772" gracePeriod=25 Nov 02 01:43:40.764559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.764521 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq"] Nov 02 01:43:40.765017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.764983 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" containerID="cri-o://0e029a7fd78c7661a5251423a0541d1436bc4c06bf9b2de8455a9d59515c86b8" gracePeriod=30 Nov 02 01:43:40.847213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.837655 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7"] Nov 02 01:43:40.847213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.838836 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" containerID="cri-o://4293a9594c01b74313af985fcdf59ce3820942aab20d50ab18d082ab2d70c294" gracePeriod=30 Nov 02 01:43:40.908699 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.908671 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:43:40.914090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:40.914067 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:43:41.023746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.023709 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume\") pod \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " Nov 02 01:43:41.024090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.024064 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume\") pod \"d72601e1-f811-4ae5-bb74-af8ced4a5363\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " Nov 02 01:43:41.024360 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.024316 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp\" (UniqueName: \"kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp\") pod \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\" (UID: \"d03985cd-4dd2-456e-9683-a4de3c0fa09c\") " Nov 02 01:43:41.024462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.024364 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume" pod "d03985cd-4dd2-456e-9683-a4de3c0fa09c" (UID: "d03985cd-4dd2-456e-9683-a4de3c0fa09c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:41.024535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.024510 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx\" (UniqueName: \"kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx\") pod \"d72601e1-f811-4ae5-bb74-af8ced4a5363\" (UID: \"d72601e1-f811-4ae5-bb74-af8ced4a5363\") " Nov 02 01:43:41.025329 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.025298 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03985cd-4dd2-456e-9683-a4de3c0fa09c-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:41.026242 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.026221 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume" pod "d72601e1-f811-4ae5-bb74-af8ced4a5363" (UID: "d72601e1-f811-4ae5-bb74-af8ced4a5363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:41.029076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.029042 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp" pod "d03985cd-4dd2-456e-9683-a4de3c0fa09c" (UID: "d03985cd-4dd2-456e-9683-a4de3c0fa09c"). InnerVolumeSpecName "kube-api-access-jpkxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:41.029880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.029846 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx" pod "d72601e1-f811-4ae5-bb74-af8ced4a5363" (UID: "d72601e1-f811-4ae5-bb74-af8ced4a5363"). InnerVolumeSpecName "kube-api-access-v7rcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:41.127198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.127168 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72601e1-f811-4ae5-bb74-af8ced4a5363-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:41.127961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.127938 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkxp\" (UniqueName: \"kubernetes.io/projected/d03985cd-4dd2-456e-9683-a4de3c0fa09c-kube-api-access-jpkxp\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:41.127961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.127961 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7rcx\" (UniqueName: \"kubernetes.io/projected/d72601e1-f811-4ae5-bb74-af8ced4a5363-kube-api-access-v7rcx\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:41.227290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.227246 4552 generic.go:356] "Generic (PLEG): container finished" podID="d72601e1-f811-4ae5-bb74-af8ced4a5363" containerID="0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772" exitCode=0 Nov 02 01:43:41.227290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.227291 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" Nov 02 01:43:41.227850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.227307 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" event={"ID":"d72601e1-f811-4ae5-bb74-af8ced4a5363","Type":"ContainerDied","Data":"0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772"} Nov 02 01:43:41.227850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.227375 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4" event={"ID":"d72601e1-f811-4ae5-bb74-af8ced4a5363","Type":"ContainerDied","Data":"68f13491cba80d926f49ef26cfe066c7c31bf149aaa8df0adb6d26fda2f32e31"} Nov 02 01:43:41.227850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.227407 4552 scope.go:122] "RemoveContainer" containerID="0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772" Nov 02 01:43:41.230256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.230208 4552 generic.go:356] "Generic (PLEG): container finished" podID="d03985cd-4dd2-456e-9683-a4de3c0fa09c" containerID="d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535" exitCode=0 Nov 02 01:43:41.230421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.230401 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" event={"ID":"d03985cd-4dd2-456e-9683-a4de3c0fa09c","Type":"ContainerDied","Data":"d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535"} Nov 02 01:43:41.230509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.230491 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" event={"ID":"d03985cd-4dd2-456e-9683-a4de3c0fa09c","Type":"ContainerDied","Data":"59281b3382e65c94935ded17856bd5f30674fee7715f2cf75e5a6ff7db01e7f9"} Nov 02 01:43:41.231196 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.230605 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr" Nov 02 01:43:41.413155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.409431 4552 scope.go:122] "RemoveContainer" containerID="0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772" Nov 02 01:43:41.454205 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:41.453779 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772\": container with ID starting with 0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772 not found: ID does not exist" containerID="0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772" Nov 02 01:43:41.454205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.453828 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772"} err="failed to get container status \"0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772\": rpc error: code = NotFound desc = could not find container \"0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772\": container with ID starting with 0248643315cd5faf24efcfc331857b8869f470705edfaa886e5da351ab396772 not found: ID does not exist" Nov 02 01:43:41.454205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.453855 4552 scope.go:122] "RemoveContainer" containerID="d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535" Nov 02 01:43:41.567703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.567673 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr"] Nov 02 01:43:41.575224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.575189 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-eeeb1a-55bbdc5fc5-dm6dr"] Nov 02 01:43:41.578489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.578472 4552 scope.go:122] "RemoveContainer" containerID="d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535" Nov 02 01:43:41.579231 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:41.579107 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535\": container with ID starting with d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535 not found: ID does not exist" containerID="d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535" Nov 02 01:43:41.579231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.579188 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535"} err="failed to get container status \"d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535\": rpc error: code = NotFound desc = could not find container \"d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535\": container with ID starting with d8a4a979353ab9c8b6f5bb5861e0d080dc7130ba54c7a1479f3c39b681d60535 not found: ID does not exist" Nov 02 01:43:41.579870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.579825 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4"] Nov 02 01:43:41.586033 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:41.586005 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-eeeb1a-5fc9c495f9-dcjh4"] Nov 02 01:43:42.049492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:42.049352 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" path="/var/lib/kubelet/pods/ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da/volumes" Nov 02 01:43:42.050084 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:42.050049 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03985cd-4dd2-456e-9683-a4de3c0fa09c" path="/var/lib/kubelet/pods/d03985cd-4dd2-456e-9683-a4de3c0fa09c/volumes" Nov 02 01:43:42.050658 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:42.050633 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72601e1-f811-4ae5-bb74-af8ced4a5363" path="/var/lib/kubelet/pods/d72601e1-f811-4ae5-bb74-af8ced4a5363/volumes" Nov 02 01:43:46.300708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.300561 4552 generic.go:356] "Generic (PLEG): container finished" podID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerID="4293a9594c01b74313af985fcdf59ce3820942aab20d50ab18d082ab2d70c294" exitCode=0 Nov 02 01:43:46.300708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.300633 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" event={"ID":"7ec77da6-3db7-4bde-951a-d829e9706a9c","Type":"ContainerDied","Data":"4293a9594c01b74313af985fcdf59ce3820942aab20d50ab18d082ab2d70c294"} Nov 02 01:43:46.302868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.302843 4552 generic.go:356] "Generic (PLEG): container finished" podID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerID="0e029a7fd78c7661a5251423a0541d1436bc4c06bf9b2de8455a9d59515c86b8" exitCode=0 Nov 02 01:43:46.302981 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.302904 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" event={"ID":"18df2e17-68a5-45d6-826e-abaca94c16b5","Type":"ContainerDied","Data":"0e029a7fd78c7661a5251423a0541d1436bc4c06bf9b2de8455a9d59515c86b8"} Nov 02 01:43:46.492831 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.492760 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:43:46.535190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.531922 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.572857 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573002 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573055 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573090 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573137 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573210 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573275 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573314 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573363 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573401 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573428 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573452 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573513 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573542 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573580 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573633 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573732 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95\") pod \"18df2e17-68a5-45d6-826e-abaca94c16b5\" (UID: \"18df2e17-68a5-45d6-826e-abaca94c16b5\") " Nov 02 01:43:46.575396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.573842 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy\") pod \"7ec77da6-3db7-4bde-951a-d829e9706a9c\" (UID: \"7ec77da6-3db7-4bde-951a-d829e9706a9c\") " Nov 02 01:43:46.581550 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.581510 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.584992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.584954 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.585412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.585384 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.585636 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.585611 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.585827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.585806 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.586002 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.585981 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.586300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.586274 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:46.586652 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.586611 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:46.589446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.589275 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.589671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.589638 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.591657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.591631 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "kube-api-access-6ww95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:46.593583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.593540 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "kube-api-access-4t4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:46.593583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.593560 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:46.593754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.593687 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.593972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.593905 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:46.594348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.594311 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:46.595384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.595365 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token" pod "7ec77da6-3db7-4bde-951a-d829e9706a9c" (UID: "7ec77da6-3db7-4bde-951a-d829e9706a9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:46.595904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.595864 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token" pod "18df2e17-68a5-45d6-826e-abaca94c16b5" (UID: "18df2e17-68a5-45d6-826e-abaca94c16b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:46.676544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676493 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ww95\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-kube-api-access-6ww95\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676537 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676554 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676568 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676580 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676596 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676609 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676623 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676637 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676651 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18df2e17-68a5-45d6-826e-abaca94c16b5-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676663 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676676 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18df2e17-68a5-45d6-826e-abaca94c16b5-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676690 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676703 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676717 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18df2e17-68a5-45d6-826e-abaca94c16b5-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676756 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec77da6-3db7-4bde-951a-d829e9706a9c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676771 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t4nh\" (UniqueName: \"kubernetes.io/projected/7ec77da6-3db7-4bde-951a-d829e9706a9c-kube-api-access-4t4nh\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:46.676806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:46.676785 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec77da6-3db7-4bde-951a-d829e9706a9c-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.312614 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.312584 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" Nov 02 01:43:47.312614 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.312603 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7" event={"ID":"7ec77da6-3db7-4bde-951a-d829e9706a9c","Type":"ContainerDied","Data":"e391274b9434a870eefb70602b12dd7840f1df6d5f48cdc149c7f728425e5f89"} Nov 02 01:43:47.313209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.312665 4552 scope.go:122] "RemoveContainer" containerID="4293a9594c01b74313af985fcdf59ce3820942aab20d50ab18d082ab2d70c294" Nov 02 01:43:47.318476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.318451 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" Nov 02 01:43:47.318612 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.318573 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq" event={"ID":"18df2e17-68a5-45d6-826e-abaca94c16b5","Type":"ContainerDied","Data":"e66b2f41cf265d3cee89ed51d041649d72569142aace3441106b15018e2cb120"} Nov 02 01:43:47.327796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.327771 4552 scope.go:122] "RemoveContainer" containerID="0e029a7fd78c7661a5251423a0541d1436bc4c06bf9b2de8455a9d59515c86b8" Nov 02 01:43:47.344972 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.344945 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7"] Nov 02 01:43:47.350775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.350742 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-k6cb7"] Nov 02 01:43:47.360470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.360440 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq"] Nov 02 01:43:47.369334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:47.365763 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8xgvq"] Nov 02 01:43:48.050432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:48.050394 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" path="/var/lib/kubelet/pods/18df2e17-68a5-45d6-826e-abaca94c16b5/volumes" Nov 02 01:43:48.051070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:48.051045 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" path="/var/lib/kubelet/pods/7ec77da6-3db7-4bde-951a-d829e9706a9c/volumes" Nov 02 01:43:55.497912 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:55.497876 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p"] Nov 02 01:43:55.498372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:55.498132 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" containerID="cri-o://853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42" gracePeriod=30 Nov 02 01:43:55.964394 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:55.964219 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr"] Nov 02 01:43:55.964574 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:55.964501 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" containerID="cri-o://0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797" gracePeriod=30 Nov 02 01:43:57.688490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:57.688456 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc"] Nov 02 01:43:57.688915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:57.688710 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" podUID="bfa01645-9018-4903-b68a-5afd9ca285ac" containerName="maas-api" containerID="cri-o://c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b" gracePeriod=30 Nov 02 01:43:57.761056 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:57.761000 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:43:58.380752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.380726 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:43:58.409342 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.409308 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs\") pod \"bfa01645-9018-4903-b68a-5afd9ca285ac\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " Nov 02 01:43:58.409529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.409370 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w\" (UniqueName: \"kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w\") pod \"bfa01645-9018-4903-b68a-5afd9ca285ac\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " Nov 02 01:43:58.409529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.409518 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\") pod \"bfa01645-9018-4903-b68a-5afd9ca285ac\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " Nov 02 01:43:58.413527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.413493 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs" pod "bfa01645-9018-4903-b68a-5afd9ca285ac" (UID: "bfa01645-9018-4903-b68a-5afd9ca285ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:58.413626 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.413555 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w" pod "bfa01645-9018-4903-b68a-5afd9ca285ac" (UID: "bfa01645-9018-4903-b68a-5afd9ca285ac"). InnerVolumeSpecName "kube-api-access-k6f9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:58.413833 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.413816 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls" pod "bfa01645-9018-4903-b68a-5afd9ca285ac" (UID: "bfa01645-9018-4903-b68a-5afd9ca285ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:58.465144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.464470 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" Nov 02 01:43:58.465144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.464491 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" event={"ID":"bfa01645-9018-4903-b68a-5afd9ca285ac","Type":"ContainerDied","Data":"c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b"} Nov 02 01:43:58.465144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.464557 4552 scope.go:122] "RemoveContainer" containerID="c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b" Nov 02 01:43:58.465144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.464927 4552 generic.go:356] "Generic (PLEG): container finished" podID="bfa01645-9018-4903-b68a-5afd9ca285ac" containerID="c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b" exitCode=0 Nov 02 01:43:58.465144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.465034 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc" event={"ID":"bfa01645-9018-4903-b68a-5afd9ca285ac","Type":"ContainerDied","Data":"06f8728a41a5b61ec9385a79ca6834d249080949cd499ba5087784b27944f3c5"} Nov 02 01:43:58.481705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.481675 4552 scope.go:122] "RemoveContainer" containerID="c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b" Nov 02 01:43:58.483021 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:43:58.482221 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b\": container with ID starting with c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b not found: ID does not exist" containerID="c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b" Nov 02 01:43:58.483021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.482260 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b"} err="failed to get container status \"c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b\": rpc error: code = NotFound desc = could not find container \"c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b\": container with ID starting with c3ce0355626bec25c1f1d4f41efc16e368dd9e769e2e6c0de1a8bb6415f9c92b not found: ID does not exist" Nov 02 01:43:58.511217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.511192 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca\") pod \"bfa01645-9018-4903-b68a-5afd9ca285ac\" (UID: \"bfa01645-9018-4903-b68a-5afd9ca285ac\") " Nov 02 01:43:58.511636 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.511593 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca" pod "bfa01645-9018-4903-b68a-5afd9ca285ac" (UID: "bfa01645-9018-4903-b68a-5afd9ca285ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:58.511964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.511944 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6f9w\" (UniqueName: \"kubernetes.io/projected/bfa01645-9018-4903-b68a-5afd9ca285ac-kube-api-access-k6f9w\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:58.512032 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.511972 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:58.512032 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.511985 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa01645-9018-4903-b68a-5afd9ca285ac-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:58.512032 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:58.511999 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa01645-9018-4903-b68a-5afd9ca285ac-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:59.003231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:59.003187 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc"] Nov 02 01:43:59.006993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:43:59.006963 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffe5c1-657d5bf7c7-nk4rc"] Nov 02 01:44:00.050260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.050217 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa01645-9018-4903-b68a-5afd9ca285ac" path="/var/lib/kubelet/pods/bfa01645-9018-4903-b68a-5afd9ca285ac/volumes" Nov 02 01:44:00.797715 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.797679 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:44:00.849490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849460 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849504 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849587 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849650 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849680 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849715 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849791 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.849950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.849839 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.850154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850136 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data\") pod \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\" (UID: \"4598ffff-e996-43b2-b1cd-c87e754d3cfd\") " Nov 02 01:44:00.850303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850157 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:00.850359 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850338 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:00.850535 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850513 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:00.850918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850858 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:00.850918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850888 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:00.850918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850966 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.850918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.850989 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.850918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.851003 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.853962 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.853907 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:00.854074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.853954 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:00.854074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.853989 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "kube-api-access-9c8pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:00.854644 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.854621 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token" pod "4598ffff-e996-43b2-b1cd-c87e754d3cfd" (UID: "4598ffff-e996-43b2-b1cd-c87e754d3cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:00.952687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.952655 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.952687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.952687 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.952960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.952727 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.952960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.952743 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8pt\" (UniqueName: \"kubernetes.io/projected/4598ffff-e996-43b2-b1cd-c87e754d3cfd-kube-api-access-9c8pt\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.952960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.952758 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:00.952960 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:00.952771 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4598ffff-e996-43b2-b1cd-c87e754d3cfd-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.173070 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.173037 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9"] Nov 02 01:44:01.173530 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.173304 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" podUID="a193d88f-a4a0-4def-95d8-2087cbc94d02" containerName="maas-api" containerID="cri-o://774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e" gracePeriod=30 Nov 02 01:44:01.297403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.297381 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362018 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362179 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362210 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362277 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362330 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362374 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362592 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362657 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.363904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.362700 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo\") pod \"7ee47da7-374c-4419-a443-013a36e93790\" (UID: \"7ee47da7-374c-4419-a443-013a36e93790\") " Nov 02 01:44:01.364842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.364818 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:01.365284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.365235 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.365573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.365437 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.365573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.365545 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.366037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.366007 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.366366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.366340 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:01.367525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.367369 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:01.368779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.368757 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "kube-api-access-9ncln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:01.369885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.369852 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy" pod "7ee47da7-374c-4419-a443-013a36e93790" (UID: "7ee47da7-374c-4419-a443-013a36e93790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478557 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478599 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478615 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478630 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ncln\" (UniqueName: \"kubernetes.io/projected/7ee47da7-374c-4419-a443-013a36e93790-kube-api-access-9ncln\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478649 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478663 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee47da7-374c-4419-a443-013a36e93790-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478677 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee47da7-374c-4419-a443-013a36e93790-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478690 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.494197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.478703 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee47da7-374c-4419-a443-013a36e93790-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.554410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.542244 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:44:01.612201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.586884 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls\") pod \"a193d88f-a4a0-4def-95d8-2087cbc94d02\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " Nov 02 01:44:01.612201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.587092 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs\") pod \"a193d88f-a4a0-4def-95d8-2087cbc94d02\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " Nov 02 01:44:01.612201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.587212 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca\") pod \"a193d88f-a4a0-4def-95d8-2087cbc94d02\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " Nov 02 01:44:01.612201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.587353 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz\" (UniqueName: \"kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz\") pod \"a193d88f-a4a0-4def-95d8-2087cbc94d02\" (UID: \"a193d88f-a4a0-4def-95d8-2087cbc94d02\") " Nov 02 01:44:01.612201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.591987 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca" pod "a193d88f-a4a0-4def-95d8-2087cbc94d02" (UID: "a193d88f-a4a0-4def-95d8-2087cbc94d02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:01.620489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.619800 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls" pod "a193d88f-a4a0-4def-95d8-2087cbc94d02" (UID: "a193d88f-a4a0-4def-95d8-2087cbc94d02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:01.639143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.635282 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs" pod "a193d88f-a4a0-4def-95d8-2087cbc94d02" (UID: "a193d88f-a4a0-4def-95d8-2087cbc94d02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:01.661509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.655816 4552 generic.go:356] "Generic (PLEG): container finished" podID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerID="853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42" exitCode=0 Nov 02 01:44:01.661509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.655928 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" event={"ID":"4598ffff-e996-43b2-b1cd-c87e754d3cfd","Type":"ContainerDied","Data":"853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42"} Nov 02 01:44:01.661509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.655979 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" event={"ID":"4598ffff-e996-43b2-b1cd-c87e754d3cfd","Type":"ContainerDied","Data":"948e27cb6844baa0dbb2a2c6a1c7b65f221991ea2f396e5c91bfe6a2423231e9"} Nov 02 01:44:01.661509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.656006 4552 scope.go:122] "RemoveContainer" containerID="853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42" Nov 02 01:44:01.661509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.656181 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p" Nov 02 01:44:01.672018 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.671692 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" Nov 02 01:44:01.672018 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.671942 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" event={"ID":"a193d88f-a4a0-4def-95d8-2087cbc94d02","Type":"ContainerDied","Data":"774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e"} Nov 02 01:44:01.672446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.671544 4552 generic.go:356] "Generic (PLEG): container finished" podID="a193d88f-a4a0-4def-95d8-2087cbc94d02" containerID="774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e" exitCode=0 Nov 02 01:44:01.672717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.672645 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9" event={"ID":"a193d88f-a4a0-4def-95d8-2087cbc94d02","Type":"ContainerDied","Data":"8e332234dd121ce25d80637b7647d08bde7c279f9792641d0c6e71a7d0ac58d3"} Nov 02 01:44:01.675132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.675089 4552 generic.go:356] "Generic (PLEG): container finished" podID="7ee47da7-374c-4419-a443-013a36e93790" containerID="0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797" exitCode=0 Nov 02 01:44:01.675367 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.675345 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" event={"ID":"7ee47da7-374c-4419-a443-013a36e93790","Type":"ContainerDied","Data":"0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797"} Nov 02 01:44:01.675449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.675397 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" event={"ID":"7ee47da7-374c-4419-a443-013a36e93790","Type":"ContainerDied","Data":"a24d06c40b3ef01a488fea557ae69c6ef34620a3aaf27aaf5eadd5606ca53764"} Nov 02 01:44:01.675507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.675450 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz" pod "a193d88f-a4a0-4def-95d8-2087cbc94d02" (UID: "a193d88f-a4a0-4def-95d8-2087cbc94d02"). InnerVolumeSpecName "kube-api-access-7lpwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:01.675507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.675474 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr" Nov 02 01:44:01.732500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.732403 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a193d88f-a4a0-4def-95d8-2087cbc94d02-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.732500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.732444 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpwz\" (UniqueName: \"kubernetes.io/projected/a193d88f-a4a0-4def-95d8-2087cbc94d02-kube-api-access-7lpwz\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.732500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.732461 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.732500 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.732474 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a193d88f-a4a0-4def-95d8-2087cbc94d02-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.781323 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.779512 4552 scope.go:122] "RemoveContainer" containerID="853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42" Nov 02 01:44:01.781323 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.779976 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p"] Nov 02 01:44:01.788997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.786456 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-cc98p"] Nov 02 01:44:01.788997 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:01.787636 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42\": container with ID starting with 853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42 not found: ID does not exist" containerID="853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42" Nov 02 01:44:01.788997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.787674 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42"} err="failed to get container status \"853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42\": rpc error: code = NotFound desc = could not find container \"853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42\": container with ID starting with 853c8b54e9b0e44e911320529aa0acde486a76baea877855aceaee81c3717b42 not found: ID does not exist" Nov 02 01:44:01.788997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.787715 4552 scope.go:122] "RemoveContainer" containerID="774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e" Nov 02 01:44:01.806274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.806249 4552 scope.go:122] "RemoveContainer" containerID="774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e" Nov 02 01:44:01.806654 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:01.806624 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e\": container with ID starting with 774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e not found: ID does not exist" containerID="774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e" Nov 02 01:44:01.806733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.806666 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e"} err="failed to get container status \"774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e\": rpc error: code = NotFound desc = could not find container \"774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e\": container with ID starting with 774e72145263a079ca5e275ce02433e7df2fd03016f2e6a355e48cb42833236e not found: ID does not exist" Nov 02 01:44:01.806733 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.806690 4552 scope.go:122] "RemoveContainer" containerID="0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797" Nov 02 01:44:01.810575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.810552 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr"] Nov 02 01:44:01.815450 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.815425 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-kfgmr"] Nov 02 01:44:01.820203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.820180 4552 scope.go:122] "RemoveContainer" containerID="0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797" Nov 02 01:44:01.820487 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:01.820470 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797\": container with ID starting with 0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797 not found: ID does not exist" containerID="0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797" Nov 02 01:44:01.820536 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.820495 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797"} err="failed to get container status \"0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797\": rpc error: code = NotFound desc = could not find container \"0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797\": container with ID starting with 0fc8e7d1c8f64cc329e1d3758b84b81998d83eff89f84462e28deb52d6afd797 not found: ID does not exist" Nov 02 01:44:01.999516 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:01.999481 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9"] Nov 02 01:44:02.003676 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:02.003655 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c22a0d-dcdbcd757-pznc9"] Nov 02 01:44:02.047520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:02.047492 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" path="/var/lib/kubelet/pods/4598ffff-e996-43b2-b1cd-c87e754d3cfd/volumes" Nov 02 01:44:02.048002 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:02.047988 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee47da7-374c-4419-a443-013a36e93790" path="/var/lib/kubelet/pods/7ee47da7-374c-4419-a443-013a36e93790/volumes" Nov 02 01:44:02.048439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:02.048426 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a193d88f-a4a0-4def-95d8-2087cbc94d02" path="/var/lib/kubelet/pods/a193d88f-a4a0-4def-95d8-2087cbc94d02/volumes" Nov 02 01:44:03.004645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:03.004611 4552 scope.go:122] "RemoveContainer" containerID="adf10848209418d4380ee774f22066c71e25baeae365b1e6f8ed55bac8ab0b91" Nov 02 01:44:16.214508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.213377 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk"] Nov 02 01:44:16.214508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.213753 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" podUID="4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" containerName="maas-api" containerID="cri-o://6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081" gracePeriod=30 Nov 02 01:44:16.740455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.740429 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:44:16.806015 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.805978 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs\") pod \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " Nov 02 01:44:16.806227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.806025 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca\") pod \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " Nov 02 01:44:16.806227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.806073 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls\") pod \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " Nov 02 01:44:16.806227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.806095 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54\" (UniqueName: \"kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54\") pod \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\" (UID: \"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1\") " Nov 02 01:44:16.806479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.806450 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca" pod "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" (UID: "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:16.807154 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.807132 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.808524 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.808495 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs" pod "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" (UID: "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:16.808716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.808698 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls" pod "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" (UID: "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:16.809357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.809337 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54" pod "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" (UID: "4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1"). InnerVolumeSpecName "kube-api-access-68k54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:16.853222 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.853186 4552 generic.go:356] "Generic (PLEG): container finished" podID="4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" containerID="6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081" exitCode=0 Nov 02 01:44:16.853389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.853238 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" Nov 02 01:44:16.853389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.853274 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" event={"ID":"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1","Type":"ContainerDied","Data":"6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081"} Nov 02 01:44:16.853389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.853344 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk" event={"ID":"4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1","Type":"ContainerDied","Data":"099dfbdb616a4d50cfeb69de5285a3e56d054c21258f8b05a78c20a1163448b1"} Nov 02 01:44:16.853389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.853373 4552 scope.go:122] "RemoveContainer" containerID="6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081" Nov 02 01:44:16.865764 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.865739 4552 scope.go:122] "RemoveContainer" containerID="6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081" Nov 02 01:44:16.866100 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:16.866043 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081\": container with ID starting with 6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081 not found: ID does not exist" containerID="6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081" Nov 02 01:44:16.866100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.866079 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081"} err="failed to get container status \"6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081\": rpc error: code = NotFound desc = could not find container \"6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081\": container with ID starting with 6170ba90e3f93528adad0907fdabbcdb3b6a2a22583d106757dbbd04ec24b081 not found: ID does not exist" Nov 02 01:44:16.883780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.883748 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk"] Nov 02 01:44:16.890768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.890737 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47dd19-5b877568b4-4tvsk"] Nov 02 01:44:16.908732 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.908685 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.908732 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.908719 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.908732 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:16.908737 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68k54\" (UniqueName: \"kubernetes.io/projected/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1-kube-api-access-68k54\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:18.106686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:18.106637 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" path="/var/lib/kubelet/pods/4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1/volumes" Nov 02 01:44:22.067560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.067501 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz"] Nov 02 01:44:22.068684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068643 4552 cpu_manager.go:414] "containerMap: removing container" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" Nov 02 01:44:22.068684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068670 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068694 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d03985cd-4dd2-456e-9683-a4de3c0fa09c" containerName="payload-processing" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068702 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03985cd-4dd2-456e-9683-a4de3c0fa09c" containerName="payload-processing" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068713 4552 cpu_manager.go:414] "containerMap: removing container" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068722 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068735 4552 cpu_manager.go:414] "containerMap: removing container" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068741 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068766 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d72601e1-f811-4ae5-bb74-af8ced4a5363" containerName="payload-pre-processing" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068773 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72601e1-f811-4ae5-bb74-af8ced4a5363" containerName="payload-pre-processing" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068787 4552 cpu_manager.go:414] "containerMap: removing container" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068795 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068808 4552 cpu_manager.go:414] "containerMap: removing container" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068815 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068853 4552 cpu_manager.go:414] "containerMap: removing container" podUID="fd7dce9b-fb14-4c23-8960-11ea4bcd109a" containerName="maas-api" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068860 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7dce9b-fb14-4c23-8960-11ea4bcd109a" containerName="maas-api" Nov 02 01:44:22.068870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068875 4552 cpu_manager.go:414] "containerMap: removing container" podUID="4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068882 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068897 4552 cpu_manager.go:414] "containerMap: removing container" podUID="a193d88f-a4a0-4def-95d8-2087cbc94d02" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068903 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a193d88f-a4a0-4def-95d8-2087cbc94d02" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068918 4552 cpu_manager.go:414] "containerMap: removing container" podUID="bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068924 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068937 4552 cpu_manager.go:414] "containerMap: removing container" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068943 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068952 4552 cpu_manager.go:414] "containerMap: removing container" podUID="946026f3-a365-45bd-9c13-98c484fd7080" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068959 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946026f3-a365-45bd-9c13-98c484fd7080" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068975 4552 cpu_manager.go:414] "containerMap: removing container" podUID="bfa01645-9018-4903-b68a-5afd9ca285ac" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068982 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa01645-9018-4903-b68a-5afd9ca285ac" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.068997 4552 cpu_manager.go:414] "containerMap: removing container" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069004 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069018 4552 cpu_manager.go:414] "containerMap: removing container" podUID="59fc09d5-d280-43af-b1aa-a09cdda17540" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069025 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fc09d5-d280-43af-b1aa-a09cdda17540" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069037 4552 cpu_manager.go:414] "containerMap: removing container" podUID="ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069043 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069061 4552 cpu_manager.go:414] "containerMap: removing container" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069068 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069250 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa01645-9018-4903-b68a-5afd9ca285ac" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069265 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee47da7-374c-4419-a443-013a36e93790" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069279 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fc09d5-d280-43af-b1aa-a09cdda17540" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069286 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8c89bf-d8ca-45c5-8d6b-8ddac25fc0da" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069298 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1ffeff-fbf6-4319-b0ae-2f934090af33" containerName="main" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069306 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="4598ffff-e996-43b2-b1cd-c87e754d3cfd" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069317 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="01903806-7eda-48ca-8097-c71608b6920f" containerName="main" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069325 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03985cd-4dd2-456e-9683-a4de3c0fa09c" containerName="payload-processing" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069334 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb503526-1cc8-4ba0-bbf2-701c6f37a0db" containerName="main" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069345 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72601e1-f811-4ae5-bb74-af8ced4a5363" containerName="payload-pre-processing" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069354 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e19217-f9be-4f7f-b790-605fe7baba8d" containerName="main" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069364 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="18df2e17-68a5-45d6-826e-abaca94c16b5" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069380 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7dce9b-fb14-4c23-8960-11ea4bcd109a" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069391 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de0b2fd-ecb6-4f9b-91d4-bd4542f077e1" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069399 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="a193d88f-a4a0-4def-95d8-2087cbc94d02" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069409 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4cdeea-aeaa-4a74-abee-8d3925ea93c8" containerName="maas-api" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069417 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec77da6-3db7-4bde-951a-d829e9706a9c" containerName="istio-proxy" Nov 02 01:44:22.069924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.069427 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="946026f3-a365-45bd-9c13-98c484fd7080" containerName="maas-api" Nov 02 01:44:22.086787 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.086672 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz"] Nov 02 01:44:22.086966 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.086795 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.096727 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.096651 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-677312\"/\"openshift-service-ca.crt\"" Nov 02 01:44:22.097040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.096816 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-677312\"/\"auth-test-model-677312-kserve-self-signed-certs\"" Nov 02 01:44:22.097040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.096899 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-677312\"/\"default-dockercfg-n7wz9\"" Nov 02 01:44:22.097040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.096964 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-677312\"/\"kube-root-ca.crt\"" Nov 02 01:44:22.233943 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.233910 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6md\" (UniqueName: \"kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.234145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.233969 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.234198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.234172 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.234308 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.234287 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.234447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.234435 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.234485 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.234462 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.336575 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.336639 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.336721 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6md\" (UniqueName: \"kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.336782 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.336909 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.336977 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.340207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.337563 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.364209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.362539 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.364209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.363206 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.364209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.363314 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.384141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.373621 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.406153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.387747 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6md\" (UniqueName: \"kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md\") pod \"auth-test-model-677312-kserve-ff97cd7b5-9x9fz\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.406752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.406719 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:22.828959 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:44:22.828922 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd9253b_82f8_4930_91c8_9b2ff0cabce2.slice/crio-5cae687d674954c5fb5de17af154b2d6cb966ec984c704ae153cdb7c9cf94121 WatchSource:0}: Error finding container 5cae687d674954c5fb5de17af154b2d6cb966ec984c704ae153cdb7c9cf94121: Status 404 returned error can't find the container with id 5cae687d674954c5fb5de17af154b2d6cb966ec984c704ae153cdb7c9cf94121 Nov 02 01:44:22.829151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.829109 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz"] Nov 02 01:44:22.978498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.978462 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq"] Nov 02 01:44:22.992043 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.992010 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" event={"ID":"cbd9253b-82f8-4930-91c8-9b2ff0cabce2","Type":"ContainerStarted","Data":"5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756"} Nov 02 01:44:22.992249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:22.992057 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" event={"ID":"cbd9253b-82f8-4930-91c8-9b2ff0cabce2","Type":"ContainerStarted","Data":"5cae687d674954c5fb5de17af154b2d6cb966ec984c704ae153cdb7c9cf94121"} Nov 02 01:44:23.025224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:23.025170 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" podStartSLOduration=1.025111147 podStartE2EDuration="1.025111147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:23.019067818 +0000 UTC m=+1763.124856726" watchObservedRunningTime="2026-11-02 01:44:23.025111147 +0000 UTC m=+1763.130900060" Nov 02 01:44:23.407849 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:23.407818 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:25.522089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:25.521793 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd"] Nov 02 01:44:25.522089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:25.522055 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" containerID="cri-o://02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e" gracePeriod=30 Nov 02 01:44:25.528527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:25.528499 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw"] Nov 02 01:44:25.528718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:25.528696 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" containerID="cri-o://62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845" gracePeriod=30 Nov 02 01:44:27.080338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:27.080295 4552 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:44:27.080810 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:27.080367 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:44:27.754452 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:27.754407 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:44:29.975721 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:29.975688 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:44:30.869980 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.869934 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:44:30.920618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.920588 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:44:30.982369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982334 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982437 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982471 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982511 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982544 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982687 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982726 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982779 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982815 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.982947 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token\") pod \"debef081-f6cd-4537-bc41-e1644e0c5200\" (UID: \"debef081-f6cd-4537-bc41-e1644e0c5200\") " Nov 02 01:44:30.983546 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.983090 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.984366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.983621 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.984366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.983851 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debef081-f6cd-4537-bc41-e1644e0c5200-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:30.984366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.983872 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:30.984366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.983887 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:30.984366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.984094 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.984769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.984736 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.986239 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.986209 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "kube-api-access-26b96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:30.986849 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.986823 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:30.987532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.987510 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:30.987915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:30.987899 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy" pod "debef081-f6cd-4537-bc41-e1644e0c5200" (UID: "debef081-f6cd-4537-bc41-e1644e0c5200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.095979 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096097 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096152 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096193 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096346 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096400 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096562 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096617 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.096686 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket\") pod \"f1139d23-22e0-4437-be47-a1f225d775b9\" (UID: \"f1139d23-22e0-4437-be47-a1f225d775b9\") " Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.097392 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26b96\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-kube-api-access-26b96\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.097416 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.097432 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.097450 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debef081-f6cd-4537-bc41-e1644e0c5200-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.097464 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debef081-f6cd-4537-bc41-e1644e0c5200-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.100201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.097477 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debef081-f6cd-4537-bc41-e1644e0c5200-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.117954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.117315 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.117954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.117581 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.123560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.123222 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.143411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.143353 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.157206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.144383 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:31.161522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.161383 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:31.161522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.161472 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.161941 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.161899 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.167272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.166143 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv" pod "f1139d23-22e0-4437-be47-a1f225d775b9" (UID: "f1139d23-22e0-4437-be47-a1f225d775b9"). InnerVolumeSpecName "kube-api-access-xchkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.198138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.175883 4552 generic.go:356] "Generic (PLEG): container finished" podID="debef081-f6cd-4537-bc41-e1644e0c5200" containerID="02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e" exitCode=0 Nov 02 01:44:31.198138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.176108 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" event={"ID":"debef081-f6cd-4537-bc41-e1644e0c5200","Type":"ContainerDied","Data":"02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e"} Nov 02 01:44:31.198138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.176189 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" event={"ID":"debef081-f6cd-4537-bc41-e1644e0c5200","Type":"ContainerDied","Data":"be1aae800428e69d5015c6cc0d4790e25fb18ef86cee060c808d9f15fbc50908"} Nov 02 01:44:31.198138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.176220 4552 scope.go:122] "RemoveContainer" containerID="02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e" Nov 02 01:44:31.198138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.176489 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd" Nov 02 01:44:31.206684 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.206634 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.206926 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.206910 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1139d23-22e0-4437-be47-a1f225d775b9-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.207097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.207078 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1139d23-22e0-4437-be47-a1f225d775b9-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.207573 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.207555 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.208034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.207975 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.208201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.208187 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.208301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.208290 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.208398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.208376 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1139d23-22e0-4437-be47-a1f225d775b9-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.208509 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.208469 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xchkv\" (UniqueName: \"kubernetes.io/projected/f1139d23-22e0-4437-be47-a1f225d775b9-kube-api-access-xchkv\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.231657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.228401 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" event={"ID":"f1139d23-22e0-4437-be47-a1f225d775b9","Type":"ContainerDied","Data":"62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845"} Nov 02 01:44:31.231657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.228513 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" Nov 02 01:44:31.251689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.250328 4552 generic.go:356] "Generic (PLEG): container finished" podID="f1139d23-22e0-4437-be47-a1f225d775b9" containerID="62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845" exitCode=0 Nov 02 01:44:31.251689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.250614 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw" event={"ID":"f1139d23-22e0-4437-be47-a1f225d775b9","Type":"ContainerDied","Data":"a39e8644db45acbb7335a4e874814b8d0e9a9b731e3161aa6f444280d1ddc466"} Nov 02 01:44:31.294374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.294295 4552 scope.go:122] "RemoveContainer" containerID="02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e" Nov 02 01:44:31.294868 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:31.294817 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e\": container with ID starting with 02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e not found: ID does not exist" containerID="02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e" Nov 02 01:44:31.294973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.294881 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e"} err="failed to get container status \"02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e\": rpc error: code = NotFound desc = could not find container \"02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e\": container with ID starting with 02ace257c3179275bbcefb643b36df69a7ae24e589102dcf65638d3b6f18026e not found: ID does not exist" Nov 02 01:44:31.294973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.294906 4552 scope.go:122] "RemoveContainer" containerID="62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845" Nov 02 01:44:31.299812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.299738 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd"] Nov 02 01:44:31.306042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.306011 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-p9xfd"] Nov 02 01:44:31.311800 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.311777 4552 scope.go:122] "RemoveContainer" containerID="62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845" Nov 02 01:44:31.316898 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:31.316871 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845\": container with ID starting with 62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845 not found: ID does not exist" containerID="62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845" Nov 02 01:44:31.323378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.323341 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845"} err="failed to get container status \"62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845\": rpc error: code = NotFound desc = could not find container \"62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845\": container with ID starting with 62e93187406cee2dd4c61139c970b1ec50d6ba317106efdf482cf43d0dfe4845 not found: ID does not exist" Nov 02 01:44:31.323867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.322527 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw"] Nov 02 01:44:31.329168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:31.326567 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-8z8jw"] Nov 02 01:44:32.046594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:32.046560 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" path="/var/lib/kubelet/pods/debef081-f6cd-4537-bc41-e1644e0c5200/volumes" Nov 02 01:44:32.047077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:32.047063 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" path="/var/lib/kubelet/pods/f1139d23-22e0-4437-be47-a1f225d775b9/volumes" Nov 02 01:44:32.407470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:32.407383 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:32.420359 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:32.420331 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:33.311076 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:33.311034 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:44:34.602403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:34.602362 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9"] Nov 02 01:44:38.017475 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.017427 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" podUID="2b994055-2f56-4652-978e-da8a9e5927e5" containerName="main" containerID="cri-o://3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff" gracePeriod=45 Nov 02 01:44:38.319639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.319614 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:44:38.345819 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.345780 4552 generic.go:356] "Generic (PLEG): container finished" podID="2b994055-2f56-4652-978e-da8a9e5927e5" containerID="3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff" exitCode=0 Nov 02 01:44:38.346080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.346056 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" event={"ID":"2b994055-2f56-4652-978e-da8a9e5927e5","Type":"ContainerDied","Data":"3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff"} Nov 02 01:44:38.346244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.346105 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" event={"ID":"2b994055-2f56-4652-978e-da8a9e5927e5","Type":"ContainerDied","Data":"2342a2af12482fffc35e4505d522cf0d843284e02004a41c31d0989c3657e2e4"} Nov 02 01:44:38.346244 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.346148 4552 scope.go:122] "RemoveContainer" containerID="3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff" Nov 02 01:44:38.346365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.346311 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq" Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.365190 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home\") pod \"2b994055-2f56-4652-978e-da8a9e5927e5\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.365290 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm\") pod \"2b994055-2f56-4652-978e-da8a9e5927e5\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.365339 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs\") pod \"2b994055-2f56-4652-978e-da8a9e5927e5\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.365522 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf\" (UniqueName: \"kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf\") pod \"2b994055-2f56-4652-978e-da8a9e5927e5\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.365669 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache\") pod \"2b994055-2f56-4652-978e-da8a9e5927e5\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.365734 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir\") pod \"2b994055-2f56-4652-978e-da8a9e5927e5\" (UID: \"2b994055-2f56-4652-978e-da8a9e5927e5\") " Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.366625 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir" pod "2b994055-2f56-4652-978e-da8a9e5927e5" (UID: "2b994055-2f56-4652-978e-da8a9e5927e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.366816 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home" pod "2b994055-2f56-4652-978e-da8a9e5927e5" (UID: "2b994055-2f56-4652-978e-da8a9e5927e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:38.368957 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.367680 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache" pod "2b994055-2f56-4652-978e-da8a9e5927e5" (UID: "2b994055-2f56-4652-978e-da8a9e5927e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:38.371640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.371610 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf" pod "2b994055-2f56-4652-978e-da8a9e5927e5" (UID: "2b994055-2f56-4652-978e-da8a9e5927e5"). InnerVolumeSpecName "kube-api-access-9hpgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:38.373452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.373421 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs" pod "2b994055-2f56-4652-978e-da8a9e5927e5" (UID: "2b994055-2f56-4652-978e-da8a9e5927e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:38.375040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.375013 4552 scope.go:122] "RemoveContainer" containerID="3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff" Nov 02 01:44:38.375504 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:38.375472 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff\": container with ID starting with 3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff not found: ID does not exist" containerID="3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff" Nov 02 01:44:38.375590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.375506 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff"} err="failed to get container status \"3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff\": rpc error: code = NotFound desc = could not find container \"3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff\": container with ID starting with 3b7279c832df6f24f6179cb464b593d07551acab21ad32eea7b8c8866c0049ff not found: ID does not exist" Nov 02 01:44:38.380518 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.380488 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm" pod "2b994055-2f56-4652-978e-da8a9e5927e5" (UID: "2b994055-2f56-4652-978e-da8a9e5927e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:38.488021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.487987 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hpgf\" (UniqueName: \"kubernetes.io/projected/2b994055-2f56-4652-978e-da8a9e5927e5-kube-api-access-9hpgf\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:38.488021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.488020 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:38.488302 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.488033 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:38.488302 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.488046 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:38.488302 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.488057 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b994055-2f56-4652-978e-da8a9e5927e5-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:38.488302 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.488067 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b994055-2f56-4652-978e-da8a9e5927e5-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:38.675400 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.675364 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq"] Nov 02 01:44:38.680880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:38.680852 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-41215f/test-model-41215f-kserve-7d5b454bb9-b5xqq"] Nov 02 01:44:40.050784 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:40.050737 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b994055-2f56-4652-978e-da8a9e5927e5" path="/var/lib/kubelet/pods/2b994055-2f56-4652-978e-da8a9e5927e5/volumes" Nov 02 01:44:40.336197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:40.335743 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q"] Nov 02 01:44:40.336197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:40.335999 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" containerID="cri-o://213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb" gracePeriod=30 Nov 02 01:44:45.646964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.646896 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:44:45.786286 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786251 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786306 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786356 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786479 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786527 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786594 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786587 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786722 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786802 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.786899 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786878 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket\") pod \"53142e56-8cd2-46b3-a11d-18245c3f849d\" (UID: \"53142e56-8cd2-46b3-a11d-18245c3f849d\") " Nov 02 01:44:45.787017 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.786992 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787218 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787615 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787616 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787776 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787805 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787832 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787851 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.787885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.787863 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.789258 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.789223 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:45.789577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.789549 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "kube-api-access-9dp6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:45.789814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.789783 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:45.790621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.790596 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token" pod "53142e56-8cd2-46b3-a11d-18245c3f849d" (UID: "53142e56-8cd2-46b3-a11d-18245c3f849d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:45.843438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.843402 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p"] Nov 02 01:44:45.844571 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844531 4552 cpu_manager.go:414] "containerMap: removing container" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" Nov 02 01:44:45.844571 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844556 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" Nov 02 01:44:45.844770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844595 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" Nov 02 01:44:45.844770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844603 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" Nov 02 01:44:45.844770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844632 4552 cpu_manager.go:414] "containerMap: removing container" podUID="2b994055-2f56-4652-978e-da8a9e5927e5" containerName="main" Nov 02 01:44:45.844770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844640 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b994055-2f56-4652-978e-da8a9e5927e5" containerName="main" Nov 02 01:44:45.844770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844664 4552 cpu_manager.go:414] "containerMap: removing container" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" Nov 02 01:44:45.844770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844671 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" Nov 02 01:44:45.845103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844845 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerName="istio-proxy" Nov 02 01:44:45.845103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844860 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1139d23-22e0-4437-be47-a1f225d775b9" containerName="istio-proxy" Nov 02 01:44:45.845103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844871 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b994055-2f56-4652-978e-da8a9e5927e5" containerName="main" Nov 02 01:44:45.845103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.844881 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="debef081-f6cd-4537-bc41-e1644e0c5200" containerName="istio-proxy" Nov 02 01:44:45.854510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.854344 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:45.859868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.859815 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e06f2\"/\"kube-root-ca.crt\"" Nov 02 01:44:45.860242 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.860224 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e06f2\"/\"openshift-service-ca.crt\"" Nov 02 01:44:45.860703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.860662 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e06f2\"/\"default-dockercfg-h8drb\"" Nov 02 01:44:45.861046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.861006 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e06f2\"/\"auth-test-model-0e06f2-kserve-self-signed-certs\"" Nov 02 01:44:45.861243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.861103 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p"] Nov 02 01:44:45.888958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.888918 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e56-8cd2-46b3-a11d-18245c3f849d-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.888958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.888952 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.888958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.888963 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dp6s\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-kube-api-access-9dp6s\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.889254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.888973 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.889254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.888982 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e56-8cd2-46b3-a11d-18245c3f849d-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:45.990534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.990493 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:45.990736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.990562 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49z8w\" (UniqueName: \"kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:45.990736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.990691 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:45.990809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.990747 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:45.990863 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.990847 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:45.990915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:45.990899 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.106536 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.106690 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.106736 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49z8w\" (UniqueName: \"kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.106840 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.106900 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.106931 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.108007 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.109141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.109022 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.119149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.113358 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.138887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.138847 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.150446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.150415 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.157686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.157625 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49z8w\" (UniqueName: \"kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w\") pod \"auth-test-model-0e06f2-kserve-659b494d9-g4d8p\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.174919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.174564 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:46.448130 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.448071 4552 generic.go:356] "Generic (PLEG): container finished" podID="53142e56-8cd2-46b3-a11d-18245c3f849d" containerID="213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb" exitCode=0 Nov 02 01:44:46.448334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.448184 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" Nov 02 01:44:46.448334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.448235 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" event={"ID":"53142e56-8cd2-46b3-a11d-18245c3f849d","Type":"ContainerDied","Data":"213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb"} Nov 02 01:44:46.448334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.448283 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q" event={"ID":"53142e56-8cd2-46b3-a11d-18245c3f849d","Type":"ContainerDied","Data":"686b24ad51cb1d4d10bff7af6ac07154e39540c5f783a2e902826a35795834b3"} Nov 02 01:44:46.448334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.448305 4552 scope.go:122] "RemoveContainer" containerID="213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb" Nov 02 01:44:46.460650 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.460622 4552 scope.go:122] "RemoveContainer" containerID="213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb" Nov 02 01:44:46.461198 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:46.461164 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb\": container with ID starting with 213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb not found: ID does not exist" containerID="213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb" Nov 02 01:44:46.461330 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.461206 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb"} err="failed to get container status \"213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb\": rpc error: code = NotFound desc = could not find container \"213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb\": container with ID starting with 213ab24e007e80a59012f0ebf4593429661c99fa3e390561bed8c6a01f4743cb not found: ID does not exist" Nov 02 01:44:46.478821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.478792 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q"] Nov 02 01:44:46.484134 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.484092 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-l6m5q"] Nov 02 01:44:46.535366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:46.535331 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p"] Nov 02 01:44:46.535851 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:44:46.535826 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f0a3813_6b8c_4908_8cef_9ec2621e179b.slice/crio-e09258a0c946ca46c930c35be159727639499732519039ad85028e2d33f22de7 WatchSource:0}: Error finding container e09258a0c946ca46c930c35be159727639499732519039ad85028e2d33f22de7: Status 404 returned error can't find the container with id e09258a0c946ca46c930c35be159727639499732519039ad85028e2d33f22de7 Nov 02 01:44:47.462970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:47.462927 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" event={"ID":"1f0a3813-6b8c-4908-8cef-9ec2621e179b","Type":"ContainerStarted","Data":"2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341"} Nov 02 01:44:47.462970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:47.462979 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" event={"ID":"1f0a3813-6b8c-4908-8cef-9ec2621e179b","Type":"ContainerStarted","Data":"e09258a0c946ca46c930c35be159727639499732519039ad85028e2d33f22de7"} Nov 02 01:44:47.486455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:47.486394 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" podStartSLOduration=2.486373917 podStartE2EDuration="2.486373917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:47.483836684 +0000 UTC m=+1787.589625594" watchObservedRunningTime="2026-11-02 01:44:47.486373917 +0000 UTC m=+1787.592162832" Nov 02 01:44:48.074708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:48.074661 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53142e56-8cd2-46b3-a11d-18245c3f849d" path="/var/lib/kubelet/pods/53142e56-8cd2-46b3-a11d-18245c3f849d/volumes" Nov 02 01:44:48.175390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:48.175358 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:49.635037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:49.634996 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" podUID="6cf7aa15-4f00-485a-a071-4f7257deca3f" containerName="main" containerID="cri-o://155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee" gracePeriod=45 Nov 02 01:44:49.943446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:49.943421 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:44:50.083995 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.083951 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache\") pod \"6cf7aa15-4f00-485a-a071-4f7257deca3f\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " Nov 02 01:44:50.084207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084077 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs\") pod \"6cf7aa15-4f00-485a-a071-4f7257deca3f\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " Nov 02 01:44:50.084296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084219 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache" pod "6cf7aa15-4f00-485a-a071-4f7257deca3f" (UID: "6cf7aa15-4f00-485a-a071-4f7257deca3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:50.084296 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084233 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm\") pod \"6cf7aa15-4f00-485a-a071-4f7257deca3f\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " Nov 02 01:44:50.084411 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084303 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir\") pod \"6cf7aa15-4f00-485a-a071-4f7257deca3f\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " Nov 02 01:44:50.084511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084476 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home\") pod \"6cf7aa15-4f00-485a-a071-4f7257deca3f\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " Nov 02 01:44:50.084657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084530 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn\" (UniqueName: \"kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn\") pod \"6cf7aa15-4f00-485a-a071-4f7257deca3f\" (UID: \"6cf7aa15-4f00-485a-a071-4f7257deca3f\") " Nov 02 01:44:50.084657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084584 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir" pod "6cf7aa15-4f00-485a-a071-4f7257deca3f" (UID: "6cf7aa15-4f00-485a-a071-4f7257deca3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:50.084779 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.084674 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home" pod "6cf7aa15-4f00-485a-a071-4f7257deca3f" (UID: "6cf7aa15-4f00-485a-a071-4f7257deca3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:50.085200 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.085175 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:50.085321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.085201 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:50.085321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.085216 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:50.086883 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.086857 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm" pod "6cf7aa15-4f00-485a-a071-4f7257deca3f" (UID: "6cf7aa15-4f00-485a-a071-4f7257deca3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:50.086968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.086931 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs" pod "6cf7aa15-4f00-485a-a071-4f7257deca3f" (UID: "6cf7aa15-4f00-485a-a071-4f7257deca3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:50.086968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.086951 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn" pod "6cf7aa15-4f00-485a-a071-4f7257deca3f" (UID: "6cf7aa15-4f00-485a-a071-4f7257deca3f"). InnerVolumeSpecName "kube-api-access-r6txn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:50.186520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.186426 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6txn\" (UniqueName: \"kubernetes.io/projected/6cf7aa15-4f00-485a-a071-4f7257deca3f-kube-api-access-r6txn\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:50.186520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.186454 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf7aa15-4f00-485a-a071-4f7257deca3f-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:50.186520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.186465 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf7aa15-4f00-485a-a071-4f7257deca3f-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:50.524396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.524360 4552 generic.go:356] "Generic (PLEG): container finished" podID="6cf7aa15-4f00-485a-a071-4f7257deca3f" containerID="155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee" exitCode=0 Nov 02 01:44:50.524584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.524413 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" Nov 02 01:44:50.524584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.524444 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" event={"ID":"6cf7aa15-4f00-485a-a071-4f7257deca3f","Type":"ContainerDied","Data":"155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee"} Nov 02 01:44:50.524584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.524503 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9" event={"ID":"6cf7aa15-4f00-485a-a071-4f7257deca3f","Type":"ContainerDied","Data":"52ed28a9e292d95238a186c74a122aeddb99cf706cb4785498c5140b6a6c1412"} Nov 02 01:44:50.524584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.524524 4552 scope.go:122] "RemoveContainer" containerID="155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee" Nov 02 01:44:50.578909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.578881 4552 scope.go:122] "RemoveContainer" containerID="155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee" Nov 02 01:44:50.581282 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:50.581232 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee\": container with ID starting with 155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee not found: ID does not exist" containerID="155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee" Nov 02 01:44:50.581414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.581286 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee"} err="failed to get container status \"155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee\": rpc error: code = NotFound desc = could not find container \"155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee\": container with ID starting with 155731da77f9c15dda7fb8c9cbf948084d20ca78e9c123a5bd0c296c323f7dee not found: ID does not exist" Nov 02 01:44:50.585985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.585956 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9"] Nov 02 01:44:50.593254 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:50.593230 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f689bc/test-model-f689bc-kserve-687fbbf795-fzzz9"] Nov 02 01:44:51.193750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.193697 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6"] Nov 02 01:44:51.194264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.193941 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" podUID="2fdc3bde-7cc5-4058-9f65-92000449176a" containerName="maas-api" containerID="cri-o://de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9" gracePeriod=30 Nov 02 01:44:51.488161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.488106 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:44:51.533620 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.533581 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6\" (UniqueName: \"kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6\") pod \"2fdc3bde-7cc5-4058-9f65-92000449176a\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " Nov 02 01:44:51.533796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.533687 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls\") pod \"2fdc3bde-7cc5-4058-9f65-92000449176a\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " Nov 02 01:44:51.533878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.533856 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs\") pod \"2fdc3bde-7cc5-4058-9f65-92000449176a\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " Nov 02 01:44:51.534060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.534042 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca\") pod \"2fdc3bde-7cc5-4058-9f65-92000449176a\" (UID: \"2fdc3bde-7cc5-4058-9f65-92000449176a\") " Nov 02 01:44:51.535104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.535075 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca" pod "2fdc3bde-7cc5-4058-9f65-92000449176a" (UID: "2fdc3bde-7cc5-4058-9f65-92000449176a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:51.538051 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.538022 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6" pod "2fdc3bde-7cc5-4058-9f65-92000449176a" (UID: "2fdc3bde-7cc5-4058-9f65-92000449176a"). InnerVolumeSpecName "kube-api-access-fmjq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:51.538287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.538263 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls" pod "2fdc3bde-7cc5-4058-9f65-92000449176a" (UID: "2fdc3bde-7cc5-4058-9f65-92000449176a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:51.538712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.538653 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs" pod "2fdc3bde-7cc5-4058-9f65-92000449176a" (UID: "2fdc3bde-7cc5-4058-9f65-92000449176a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:51.539095 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.539068 4552 generic.go:356] "Generic (PLEG): container finished" podID="2fdc3bde-7cc5-4058-9f65-92000449176a" containerID="de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9" exitCode=0 Nov 02 01:44:51.539307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.539285 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" event={"ID":"2fdc3bde-7cc5-4058-9f65-92000449176a","Type":"ContainerDied","Data":"de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9"} Nov 02 01:44:51.539382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.539346 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" event={"ID":"2fdc3bde-7cc5-4058-9f65-92000449176a","Type":"ContainerDied","Data":"056b6f95f8b9232651b10a1b3f25bd2673bf67f09be53bbeb6f8a7462830136a"} Nov 02 01:44:51.539441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.539379 4552 scope.go:122] "RemoveContainer" containerID="de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9" Nov 02 01:44:51.539514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.539500 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6" Nov 02 01:44:51.578600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.578578 4552 scope.go:122] "RemoveContainer" containerID="de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9" Nov 02 01:44:51.578976 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:51.578952 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9\": container with ID starting with de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9 not found: ID does not exist" containerID="de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9" Nov 02 01:44:51.579230 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.578983 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9"} err="failed to get container status \"de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9\": rpc error: code = NotFound desc = could not find container \"de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9\": container with ID starting with de207e2b503baa30e8e55e3e6c8f594a1f5b75c91350e61d024bb81abab366b9 not found: ID does not exist" Nov 02 01:44:51.593463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.593416 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6"] Nov 02 01:44:51.598477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.598456 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b6ec07-5457b57cfb-s8fx6"] Nov 02 01:44:51.654143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.646172 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:51.654143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.646213 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdc3bde-7cc5-4058-9f65-92000449176a-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:51.654143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.646230 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmjq6\" (UniqueName: \"kubernetes.io/projected/2fdc3bde-7cc5-4058-9f65-92000449176a-kube-api-access-fmjq6\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:51.654143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:51.646246 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdc3bde-7cc5-4058-9f65-92000449176a-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:52.052933 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.052897 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fdc3bde-7cc5-4058-9f65-92000449176a" path="/var/lib/kubelet/pods/2fdc3bde-7cc5-4058-9f65-92000449176a/volumes" Nov 02 01:44:52.053537 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.053514 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf7aa15-4f00-485a-a071-4f7257deca3f" path="/var/lib/kubelet/pods/6cf7aa15-4f00-485a-a071-4f7257deca3f/volumes" Nov 02 01:44:52.247477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.247445 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k"] Nov 02 01:44:52.248566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.248539 4552 cpu_manager.go:414] "containerMap: removing container" podUID="6cf7aa15-4f00-485a-a071-4f7257deca3f" containerName="main" Nov 02 01:44:52.248566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.248565 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf7aa15-4f00-485a-a071-4f7257deca3f" containerName="main" Nov 02 01:44:52.249089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.248629 4552 cpu_manager.go:414] "containerMap: removing container" podUID="2fdc3bde-7cc5-4058-9f65-92000449176a" containerName="maas-api" Nov 02 01:44:52.249089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.248681 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdc3bde-7cc5-4058-9f65-92000449176a" containerName="maas-api" Nov 02 01:44:52.249089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.248876 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdc3bde-7cc5-4058-9f65-92000449176a" containerName="maas-api" Nov 02 01:44:52.249089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.248917 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf7aa15-4f00-485a-a071-4f7257deca3f" containerName="main" Nov 02 01:44:52.254836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.254811 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.262724 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.262692 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k"] Nov 02 01:44:52.326455 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.326371 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c8d4a0-dockercfg-swkz7\"" Nov 02 01:44:52.437260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.348502 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c8d4a0\"" Nov 02 01:44:52.437260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.379226 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffd9\" (UniqueName: \"kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9\") pod \"payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.437260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.436793 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.538911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.538858 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffd9\" (UniqueName: \"kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9\") pod \"payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.539456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.539441 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.541004 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.540415 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.551895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.551868 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffd9\" (UniqueName: \"kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9\") pod \"payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.580547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.580491 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:52.864499 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:44:52.864464 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d34a683_dd9e_4bb4_8b63_f3e5249e523e.slice/crio-581eec7eefa8782706bbb480cc15e2b9c10c0506eac885f4ba0cd4db586c1eae WatchSource:0}: Error finding container 581eec7eefa8782706bbb480cc15e2b9c10c0506eac885f4ba0cd4db586c1eae: Status 404 returned error can't find the container with id 581eec7eefa8782706bbb480cc15e2b9c10c0506eac885f4ba0cd4db586c1eae Nov 02 01:44:52.865403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:52.865378 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k"] Nov 02 01:44:53.563663 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:53.563621 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" event={"ID":"4d34a683-dd9e-4bb4-8b63-f3e5249e523e","Type":"ContainerStarted","Data":"c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc"} Nov 02 01:44:53.564042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:53.563687 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" event={"ID":"4d34a683-dd9e-4bb4-8b63-f3e5249e523e","Type":"ContainerStarted","Data":"581eec7eefa8782706bbb480cc15e2b9c10c0506eac885f4ba0cd4db586c1eae"} Nov 02 01:44:53.564042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:53.563845 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:44:53.590554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:53.590495 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" podStartSLOduration=1.590480627 podStartE2EDuration="1.590480627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:53.588396973 +0000 UTC m=+1793.694185888" watchObservedRunningTime="2026-11-02 01:44:53.590480627 +0000 UTC m=+1793.696269541" Nov 02 01:44:55.519027 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:55.518995 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq"] Nov 02 01:44:55.519529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:55.519233 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" podUID="ec7cdbca-b490-4061-a4e9-eae28d7958af" containerName="maas-api" containerID="cri-o://c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357" gracePeriod=30 Nov 02 01:44:56.112416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.112386 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:44:56.175021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.174985 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:56.192251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.192222 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:56.201166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.201135 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\") pod \"ec7cdbca-b490-4061-a4e9-eae28d7958af\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " Nov 02 01:44:56.201320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.201186 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs\") pod \"ec7cdbca-b490-4061-a4e9-eae28d7958af\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " Nov 02 01:44:56.201477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.201441 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca\") pod \"ec7cdbca-b490-4061-a4e9-eae28d7958af\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " Nov 02 01:44:56.201549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.201513 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8\" (UniqueName: \"kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8\") pod \"ec7cdbca-b490-4061-a4e9-eae28d7958af\" (UID: \"ec7cdbca-b490-4061-a4e9-eae28d7958af\") " Nov 02 01:44:56.202377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.202349 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca" pod "ec7cdbca-b490-4061-a4e9-eae28d7958af" (UID: "ec7cdbca-b490-4061-a4e9-eae28d7958af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:56.203558 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.203540 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls" pod "ec7cdbca-b490-4061-a4e9-eae28d7958af" (UID: "ec7cdbca-b490-4061-a4e9-eae28d7958af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:56.203935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.203913 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs" pod "ec7cdbca-b490-4061-a4e9-eae28d7958af" (UID: "ec7cdbca-b490-4061-a4e9-eae28d7958af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:56.204181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.204153 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8" pod "ec7cdbca-b490-4061-a4e9-eae28d7958af" (UID: "ec7cdbca-b490-4061-a4e9-eae28d7958af"). InnerVolumeSpecName "kube-api-access-d2db8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:56.303952 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.303887 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:56.303952 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.303929 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7cdbca-b490-4061-a4e9-eae28d7958af-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:56.303952 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.303948 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cdbca-b490-4061-a4e9-eae28d7958af-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:56.304301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.303963 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2db8\" (UniqueName: \"kubernetes.io/projected/ec7cdbca-b490-4061-a4e9-eae28d7958af-kube-api-access-d2db8\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:56.593773 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.593739 4552 generic.go:356] "Generic (PLEG): container finished" podID="ec7cdbca-b490-4061-a4e9-eae28d7958af" containerID="c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357" exitCode=0 Nov 02 01:44:56.594275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.593814 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" Nov 02 01:44:56.594275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.593825 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" event={"ID":"ec7cdbca-b490-4061-a4e9-eae28d7958af","Type":"ContainerDied","Data":"c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357"} Nov 02 01:44:56.594275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.593880 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq" event={"ID":"ec7cdbca-b490-4061-a4e9-eae28d7958af","Type":"ContainerDied","Data":"f3b4ffd915b762e283a550ac37c902017585de8e6ae1db15eacffbfebccec191"} Nov 02 01:44:56.594275 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.593903 4552 scope.go:122] "RemoveContainer" containerID="c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357" Nov 02 01:44:56.606198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.606181 4552 scope.go:122] "RemoveContainer" containerID="c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357" Nov 02 01:44:56.606399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.606384 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:44:56.606518 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:56.606499 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357\": container with ID starting with c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357 not found: ID does not exist" containerID="c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357" Nov 02 01:44:56.606579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.606530 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357"} err="failed to get container status \"c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357\": rpc error: code = NotFound desc = could not find container \"c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357\": container with ID starting with c9b8dc1aab6cb08dc7bfdd9d0a4fc5581ec4c2bde1e0544ccc8182eb34886357 not found: ID does not exist" Nov 02 01:44:56.620149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.620092 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq"] Nov 02 01:44:56.624163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:56.624138 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-41215f-666948fd5-vkzlq"] Nov 02 01:44:57.754336 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:57.754296 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:44:58.072174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.072063 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7cdbca-b490-4061-a4e9-eae28d7958af" path="/var/lib/kubelet/pods/ec7cdbca-b490-4061-a4e9-eae28d7958af/volumes" Nov 02 01:44:58.072850 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.072828 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg"] Nov 02 01:44:58.074319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.074106 4552 cpu_manager.go:414] "containerMap: removing container" podUID="ec7cdbca-b490-4061-a4e9-eae28d7958af" containerName="maas-api" Nov 02 01:44:58.074452 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.074438 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7cdbca-b490-4061-a4e9-eae28d7958af" containerName="maas-api" Nov 02 01:44:58.074728 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.074715 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7cdbca-b490-4061-a4e9-eae28d7958af" containerName="maas-api" Nov 02 01:44:58.084250 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.080580 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg"] Nov 02 01:44:58.084250 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.080750 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.084853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.084828 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-0f32d6-serving-cert\"" Nov 02 01:44:58.128860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.128829 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.129012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.128941 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.129087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.129063 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9zg2\" (UniqueName: \"kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.129255 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.129241 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.230620 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.230579 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.230828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.230667 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.230828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.230753 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.230953 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.230864 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9zg2\" (UniqueName: \"kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.231226 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:58.231203 4552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-serving-cert: secret "maas-api-e2e-shared-b-w5-0f32d6-serving-cert" not found Nov 02 01:44:58.231413 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:44:58.231400 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls podName:0d357153-7b81-45fd-9a70-11e58828c720 nodeName:}" failed. No retries permitted until 2026-11-02 01:44:58.731378096 +0000 UTC m=+1798.837167002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls") pod "maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" (UID: "0d357153-7b81-45fd-9a70-11e58828c720") : secret "maas-api-e2e-shared-b-w5-0f32d6-serving-cert" not found Nov 02 01:44:58.233046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.232345 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.234041 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.233998 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.247480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.247452 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9zg2\" (UniqueName: \"kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.737430 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.737388 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:58.740170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:58.740140 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:59.004356 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:59.004278 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:44:59.307543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:59.307516 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg"] Nov 02 01:44:59.307981 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:44:59.307959 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d357153_7b81_45fd_9a70_11e58828c720.slice/crio-13c7b878eb25a08f2505d24ff9bed96149aa2c5987556574ef93ed83ac52f2ee WatchSource:0}: Error finding container 13c7b878eb25a08f2505d24ff9bed96149aa2c5987556574ef93ed83ac52f2ee: Status 404 returned error can't find the container with id 13c7b878eb25a08f2505d24ff9bed96149aa2c5987556574ef93ed83ac52f2ee Nov 02 01:44:59.395339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:59.388508 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf"] Nov 02 01:44:59.395339 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:59.388766 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" containerID="cri-o://8be8a5df4abbe35b624d5269026f45a9172eaf163a5cea4b98755dd842d98b5f" gracePeriod=30 Nov 02 01:44:59.689197 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:44:59.688981 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" event={"ID":"0d357153-7b81-45fd-9a70-11e58828c720","Type":"ContainerStarted","Data":"13c7b878eb25a08f2505d24ff9bed96149aa2c5987556574ef93ed83ac52f2ee"} Nov 02 01:45:00.698837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:00.698803 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" event={"ID":"0d357153-7b81-45fd-9a70-11e58828c720","Type":"ContainerStarted","Data":"af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e"} Nov 02 01:45:00.699233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:00.698899 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:45:00.721602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:00.721556 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" podStartSLOduration=2.224852301 podStartE2EDuration="2.721542989s" totalImagesPullingTime="496.690688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:00.718678812 +0000 UTC m=+1800.824467727" watchObservedRunningTime="2026-11-02 01:45:00.721542989 +0000 UTC m=+1800.827331904" Nov 02 01:45:02.908727 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:02.908688 4552 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:02.909251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:02.908766 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:04.632143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:04.629374 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:45:04.825886 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:04.825852 4552 generic.go:356] "Generic (PLEG): container finished" podID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerID="8be8a5df4abbe35b624d5269026f45a9172eaf163a5cea4b98755dd842d98b5f" exitCode=0 Nov 02 01:45:04.826132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:04.826098 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" event={"ID":"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d","Type":"ContainerDied","Data":"8be8a5df4abbe35b624d5269026f45a9172eaf163a5cea4b98755dd842d98b5f"} Nov 02 01:45:05.203929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.203903 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:45:05.259804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.259768 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.259996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.259864 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.259996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.259905 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.259996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.259982 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.260893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.260097 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.260893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.260178 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.260893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.260324 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.260893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.260387 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.260893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.260578 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:05.261282 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.261152 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.261282 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.261213 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.261465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.261443 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.265628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.265578 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.265882 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.265863 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.266314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.266289 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "kube-api-access-v274v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:05.268809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.268781 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:05.278564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.278538 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:05.361735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.361683 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket\") pod \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\" (UID: \"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d\") " Nov 02 01:45:05.362642 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362616 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362650 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v274v\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-kube-api-access-v274v\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362663 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362676 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362688 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362702 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362714 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.362982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.362963 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket" pod "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" (UID: "81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.464334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.464252 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.835298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.835265 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" Nov 02 01:45:05.835298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.835279 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf" event={"ID":"81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d","Type":"ContainerDied","Data":"254294ccddd83eee13b71ebabbe386f8476ac8480208dcda533daaaa79442c63"} Nov 02 01:45:05.835797 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.835345 4552 scope.go:122] "RemoveContainer" containerID="8be8a5df4abbe35b624d5269026f45a9172eaf163a5cea4b98755dd842d98b5f" Nov 02 01:45:05.866674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.866644 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf"] Nov 02 01:45:05.871176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:05.871150 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-4kgdf"] Nov 02 01:45:06.080464 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.080393 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" path="/var/lib/kubelet/pods/81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d/volumes" Nov 02 01:45:06.341377 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.341338 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw"] Nov 02 01:45:06.342739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.342707 4552 cpu_manager.go:414] "containerMap: removing container" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" Nov 02 01:45:06.342739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.342736 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" Nov 02 01:45:06.342988 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.342980 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e0dd6f-2ed0-4d94-9d4d-a40952a0ea2d" containerName="istio-proxy" Nov 02 01:45:06.351310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.351250 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.377069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.377037 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c8d4a0-serving-cert\"" Nov 02 01:45:06.384995 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.382763 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw"] Nov 02 01:45:06.408790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.408747 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.408982 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.408934 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.409093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.409071 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.409211 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.409191 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4k2n\" (UniqueName: \"kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.535140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.534647 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.535140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.534759 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.535140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.534825 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4k2n\" (UniqueName: \"kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.535140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.534984 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.550997 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.546839 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.570141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.566010 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.570141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.566524 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.579143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.575443 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4k2n\" (UniqueName: \"kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n\") pod \"maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.701202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.701011 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:06.729166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:06.727647 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:45:07.212234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:07.208524 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw"] Nov 02 01:45:07.936556 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:07.936520 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" event={"ID":"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5","Type":"ContainerStarted","Data":"d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae"} Nov 02 01:45:07.936756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:07.936588 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" event={"ID":"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5","Type":"ContainerStarted","Data":"89c4b2e1cdac70d88e8f4485d335fd5c02c548914e991f55f3c2c123627afc0c"} Nov 02 01:45:07.936756 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:07.936720 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:07.959252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:07.959198 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" podStartSLOduration=1.4377738660000001 podStartE2EDuration="1.959179696s" totalImagesPullingTime="521.40583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:07.956398967 +0000 UTC m=+1808.062187882" watchObservedRunningTime="2026-11-02 01:45:07.959179696 +0000 UTC m=+1808.064968849" Nov 02 01:45:08.750818 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:08.750777 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd"] Nov 02 01:45:13.985487 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:13.985416 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:45:19.722884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:19.722848 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph"] Nov 02 01:45:19.723894 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:19.723604 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" podUID="8b586b83-adea-4ef9-8aa2-76c0bc61988c" containerName="maas-api" containerID="cri-o://a07d677a25ae96efab608dae0ad5b07d3aadc0b8547be8ae4a68b5d8ac7ca7a4" gracePeriod=30 Nov 02 01:45:20.173083 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.172991 4552 generic.go:356] "Generic (PLEG): container finished" podID="8b586b83-adea-4ef9-8aa2-76c0bc61988c" containerID="a07d677a25ae96efab608dae0ad5b07d3aadc0b8547be8ae4a68b5d8ac7ca7a4" exitCode=0 Nov 02 01:45:20.178577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.178542 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" event={"ID":"8b586b83-adea-4ef9-8aa2-76c0bc61988c","Type":"ContainerDied","Data":"a07d677a25ae96efab608dae0ad5b07d3aadc0b8547be8ae4a68b5d8ac7ca7a4"} Nov 02 01:45:20.211320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.211292 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:45:20.352807 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.352100 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs\") pod \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " Nov 02 01:45:20.353165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.353141 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m\" (UniqueName: \"kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m\") pod \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " Nov 02 01:45:20.375277 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.375246 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca\") pod \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " Nov 02 01:45:20.375954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.375824 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca" pod "8b586b83-adea-4ef9-8aa2-76c0bc61988c" (UID: "8b586b83-adea-4ef9-8aa2-76c0bc61988c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:20.376427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.375979 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls\") pod \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\" (UID: \"8b586b83-adea-4ef9-8aa2-76c0bc61988c\") " Nov 02 01:45:20.377168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.377110 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b586b83-adea-4ef9-8aa2-76c0bc61988c-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:20.380891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.380847 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls" pod "8b586b83-adea-4ef9-8aa2-76c0bc61988c" (UID: "8b586b83-adea-4ef9-8aa2-76c0bc61988c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:20.382642 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.382603 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m" pod "8b586b83-adea-4ef9-8aa2-76c0bc61988c" (UID: "8b586b83-adea-4ef9-8aa2-76c0bc61988c"). InnerVolumeSpecName "kube-api-access-77g8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:20.400456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.396389 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs" pod "8b586b83-adea-4ef9-8aa2-76c0bc61988c" (UID: "8b586b83-adea-4ef9-8aa2-76c0bc61988c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:20.478585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.478555 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:20.478585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.478584 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77g8m\" (UniqueName: \"kubernetes.io/projected/8b586b83-adea-4ef9-8aa2-76c0bc61988c-kube-api-access-77g8m\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:20.478791 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:20.478598 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b586b83-adea-4ef9-8aa2-76c0bc61988c-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:21.186206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:21.186175 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" Nov 02 01:45:21.186719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:21.186174 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph" event={"ID":"8b586b83-adea-4ef9-8aa2-76c0bc61988c","Type":"ContainerDied","Data":"2c0f1b099b442056b5fc498669f714712f1c752cabacdfe07f320896755ea614"} Nov 02 01:45:21.186719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:21.186330 4552 scope.go:122] "RemoveContainer" containerID="a07d677a25ae96efab608dae0ad5b07d3aadc0b8547be8ae4a68b5d8ac7ca7a4" Nov 02 01:45:21.218563 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:21.218532 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph"] Nov 02 01:45:21.224105 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:21.224081 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f689bc-56dbb68879-w5mph"] Nov 02 01:45:22.050993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.050960 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b586b83-adea-4ef9-8aa2-76c0bc61988c" path="/var/lib/kubelet/pods/8b586b83-adea-4ef9-8aa2-76c0bc61988c/volumes" Nov 02 01:45:22.072365 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.072329 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8"] Nov 02 01:45:22.073719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.073684 4552 cpu_manager.go:414] "containerMap: removing container" podUID="8b586b83-adea-4ef9-8aa2-76c0bc61988c" containerName="maas-api" Nov 02 01:45:22.073843 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.073720 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b586b83-adea-4ef9-8aa2-76c0bc61988c" containerName="maas-api" Nov 02 01:45:22.073956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.073943 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b586b83-adea-4ef9-8aa2-76c0bc61988c" containerName="maas-api" Nov 02 01:45:22.088968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.088941 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.089920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.088865 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8"] Nov 02 01:45:22.100420 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.100393 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-eaec8c\"/\"auth-test-model-eaec8c-kserve-self-signed-certs\"" Nov 02 01:45:22.100565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.100433 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-eaec8c\"/\"kube-root-ca.crt\"" Nov 02 01:45:22.100565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.100498 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-eaec8c\"/\"openshift-service-ca.crt\"" Nov 02 01:45:22.100565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.100560 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-eaec8c\"/\"default-dockercfg-t9876\"" Nov 02 01:45:22.112326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.112297 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.112502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.112357 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.112502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.112438 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.112502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.112471 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.112702 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.112530 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npm79\" (UniqueName: \"kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.112804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.112776 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.214744 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.214705 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.214855 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.214885 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.214930 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.214953 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.214982 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npm79\" (UniqueName: \"kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.215161 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.215234 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.215910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.215668 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.217879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.217859 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.218016 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.217946 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.344205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.340956 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npm79\" (UniqueName: \"kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79\") pod \"auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.415794 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.415752 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:22.734264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:22.734235 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8"] Nov 02 01:45:22.734378 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:45:22.734269 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5b7dc8f_f7e2_4b81_a1e1_884cb20b215e.slice/crio-7d489272b91a1f003998e3e4aebe7e70614f48f9a7eb189e8fce03f02d501c4c WatchSource:0}: Error finding container 7d489272b91a1f003998e3e4aebe7e70614f48f9a7eb189e8fce03f02d501c4c: Status 404 returned error can't find the container with id 7d489272b91a1f003998e3e4aebe7e70614f48f9a7eb189e8fce03f02d501c4c Nov 02 01:45:23.209171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:23.209044 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" event={"ID":"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e","Type":"ContainerStarted","Data":"5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7"} Nov 02 01:45:23.209171 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:23.209090 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" event={"ID":"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e","Type":"ContainerStarted","Data":"7d489272b91a1f003998e3e4aebe7e70614f48f9a7eb189e8fce03f02d501c4c"} Nov 02 01:45:23.228679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:23.228616 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" podStartSLOduration=1.228600411 podStartE2EDuration="1.228600411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:23.227741259 +0000 UTC m=+1823.333530172" watchObservedRunningTime="2026-11-02 01:45:23.228600411 +0000 UTC m=+1823.334389316" Nov 02 01:45:23.416862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:23.416828 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:23.776554 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:23.776513 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" containerName="main" containerID="cri-o://335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978" gracePeriod=45 Nov 02 01:45:24.070874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.070840 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:45:24.189561 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189530 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm\") pod \"66129927-16bf-48e1-835d-f236f06dd6d5\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " Nov 02 01:45:24.189735 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189614 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir\") pod \"66129927-16bf-48e1-835d-f236f06dd6d5\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " Nov 02 01:45:24.189802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189791 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs\") pod \"66129927-16bf-48e1-835d-f236f06dd6d5\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " Nov 02 01:45:24.189891 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189871 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home\") pod \"66129927-16bf-48e1-835d-f236f06dd6d5\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " Nov 02 01:45:24.190038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189905 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p\" (UniqueName: \"kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p\") pod \"66129927-16bf-48e1-835d-f236f06dd6d5\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " Nov 02 01:45:24.190038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189912 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir" pod "66129927-16bf-48e1-835d-f236f06dd6d5" (UID: "66129927-16bf-48e1-835d-f236f06dd6d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:24.190038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.189936 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache\") pod \"66129927-16bf-48e1-835d-f236f06dd6d5\" (UID: \"66129927-16bf-48e1-835d-f236f06dd6d5\") " Nov 02 01:45:24.190274 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.190222 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home" pod "66129927-16bf-48e1-835d-f236f06dd6d5" (UID: "66129927-16bf-48e1-835d-f236f06dd6d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:24.190578 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.190330 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache" pod "66129927-16bf-48e1-835d-f236f06dd6d5" (UID: "66129927-16bf-48e1-835d-f236f06dd6d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:24.191082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.191056 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:24.191235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.191085 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:24.191235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.191101 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:24.192314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.192292 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs" pod "66129927-16bf-48e1-835d-f236f06dd6d5" (UID: "66129927-16bf-48e1-835d-f236f06dd6d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:24.192396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.192383 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p" pod "66129927-16bf-48e1-835d-f236f06dd6d5" (UID: "66129927-16bf-48e1-835d-f236f06dd6d5"). InnerVolumeSpecName "kube-api-access-6vq6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:24.192490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.192475 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm" pod "66129927-16bf-48e1-835d-f236f06dd6d5" (UID: "66129927-16bf-48e1-835d-f236f06dd6d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:24.219964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.219920 4552 generic.go:356] "Generic (PLEG): container finished" podID="66129927-16bf-48e1-835d-f236f06dd6d5" containerID="335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978" exitCode=0 Nov 02 01:45:24.222922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.220192 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" event={"ID":"66129927-16bf-48e1-835d-f236f06dd6d5","Type":"ContainerDied","Data":"335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978"} Nov 02 01:45:24.222922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.220340 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" event={"ID":"66129927-16bf-48e1-835d-f236f06dd6d5","Type":"ContainerDied","Data":"66646784d2c0f908cc84e2845aeda1928cc26103b20a72b114560c5dda4c6924"} Nov 02 01:45:24.222922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.220373 4552 scope.go:122] "RemoveContainer" containerID="335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978" Nov 02 01:45:24.222922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.220209 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd" Nov 02 01:45:24.237847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.237826 4552 scope.go:122] "RemoveContainer" containerID="335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978" Nov 02 01:45:24.238535 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:45:24.238513 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978\": container with ID starting with 335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978 not found: ID does not exist" containerID="335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978" Nov 02 01:45:24.238618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.238543 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978"} err="failed to get container status \"335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978\": rpc error: code = NotFound desc = could not find container \"335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978\": container with ID starting with 335d0d83a7ec96073fa435ab7562e1890c20521cefa0fd0b79d1cb06dcef3978 not found: ID does not exist" Nov 02 01:45:24.250878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.250849 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd"] Nov 02 01:45:24.254925 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.254901 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bf1e55/test-model-bf1e55-kserve-7cf46c9c59-sqvdd"] Nov 02 01:45:24.293066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.292979 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66129927-16bf-48e1-835d-f236f06dd6d5-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:24.293066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.293009 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66129927-16bf-48e1-835d-f236f06dd6d5-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:24.293066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:24.293022 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vq6p\" (UniqueName: \"kubernetes.io/projected/66129927-16bf-48e1-835d-f236f06dd6d5-kube-api-access-6vq6p\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:26.066909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:26.066869 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" path="/var/lib/kubelet/pods/66129927-16bf-48e1-835d-f236f06dd6d5/volumes" Nov 02 01:45:27.755313 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:45:27.755280 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:45:28.194581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.194406 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz"] Nov 02 01:45:28.194812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.194653 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" containerID="cri-o://d11056c596f14bd73e52c222653bdf3d503748b4fee140b0ea3c31d2d48a37af" gracePeriod=30 Nov 02 01:45:28.202012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.201725 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh"] Nov 02 01:45:28.202175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.202088 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" containerID="cri-o://d43fe3baddcd96c4d2b6907d29c23790e5007c79a8d27a4dea3b8e204fcdc8af" gracePeriod=30 Nov 02 01:45:28.209317 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.209289 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89"] Nov 02 01:45:28.209564 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.209540 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" containerID="cri-o://8443997b7217ef0f4c0572281d24cbffc76dc202277414cb5e12450755bef2ee" gracePeriod=30 Nov 02 01:45:28.213246 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.213212 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5"] Nov 02 01:45:28.213486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.213461 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" containerID="cri-o://ad63ca75c4ed7106d03ed3e847e0276da8797b198c2ee936ce59167688163a87" gracePeriod=30 Nov 02 01:45:28.641501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.641457 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:28.641656 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:28.641523 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:31.445948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:31.445904 4552 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:31.446456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:31.445977 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:32.419433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.416624 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:32.432304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.432277 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:32.897252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.897221 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd"] Nov 02 01:45:32.897992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.897977 4552 cpu_manager.go:414] "containerMap: removing container" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" containerName="main" Nov 02 01:45:32.897992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.897993 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" containerName="main" Nov 02 01:45:32.898153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.898137 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="66129927-16bf-48e1-835d-f236f06dd6d5" containerName="main" Nov 02 01:45:32.903685 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.903661 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:32.915914 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.915885 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3c4ea5\"/\"default-dockercfg-p7kn9\"" Nov 02 01:45:32.915914 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.915890 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3c4ea5\"/\"kube-root-ca.crt\"" Nov 02 01:45:32.916138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.915999 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3c4ea5\"/\"auth-test-model-3c4ea5-kserve-self-signed-certs\"" Nov 02 01:45:32.916138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.916042 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3c4ea5\"/\"openshift-service-ca.crt\"" Nov 02 01:45:32.921932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:32.921904 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd"] Nov 02 01:45:33.040000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.039968 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.040282 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.040028 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.040282 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.040179 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.040418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.040339 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.040418 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.040413 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxt2j\" (UniqueName: \"kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.040550 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.040531 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142193 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142156 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142258 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxt2j\" (UniqueName: \"kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142310 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142413 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142466 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142552 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142610 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142685 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.142926 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.142901 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.145181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.145158 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.145527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.145504 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.169669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.169558 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxt2j\" (UniqueName: \"kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j\") pod \"auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.225408 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.224433 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:33.423553 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.420967 4552 generic.go:356] "Generic (PLEG): container finished" podID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerID="d11056c596f14bd73e52c222653bdf3d503748b4fee140b0ea3c31d2d48a37af" exitCode=0 Nov 02 01:45:33.423553 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.421047 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" event={"ID":"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552","Type":"ContainerDied","Data":"d11056c596f14bd73e52c222653bdf3d503748b4fee140b0ea3c31d2d48a37af"} Nov 02 01:45:33.423553 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.423469 4552 generic.go:356] "Generic (PLEG): container finished" podID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerID="8443997b7217ef0f4c0572281d24cbffc76dc202277414cb5e12450755bef2ee" exitCode=0 Nov 02 01:45:33.423680 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.423574 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" event={"ID":"c9560f4a-a1ac-49c2-9829-728c386c6260","Type":"ContainerDied","Data":"8443997b7217ef0f4c0572281d24cbffc76dc202277414cb5e12450755bef2ee"} Nov 02 01:45:33.430295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.426850 4552 generic.go:356] "Generic (PLEG): container finished" podID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerID="ad63ca75c4ed7106d03ed3e847e0276da8797b198c2ee936ce59167688163a87" exitCode=0 Nov 02 01:45:33.430295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.427036 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" event={"ID":"fe252814-5eec-4e00-9e30-013c83dbd6d0","Type":"ContainerDied","Data":"ad63ca75c4ed7106d03ed3e847e0276da8797b198c2ee936ce59167688163a87"} Nov 02 01:45:33.430295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.428890 4552 generic.go:356] "Generic (PLEG): container finished" podID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerID="d43fe3baddcd96c4d2b6907d29c23790e5007c79a8d27a4dea3b8e204fcdc8af" exitCode=0 Nov 02 01:45:33.430295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.428963 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" event={"ID":"6f6519d4-abe0-4ce9-bc38-38993a7ad791","Type":"ContainerDied","Data":"d43fe3baddcd96c4d2b6907d29c23790e5007c79a8d27a4dea3b8e204fcdc8af"} Nov 02 01:45:33.440198 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.439480 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:45:33.604757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.604728 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:45:33.605824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.605796 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd"] Nov 02 01:45:33.617935 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:45:33.617903 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7add8ae_4a07_44e6_ae69_be43510e6dce.slice/crio-08d0099708d66788226d18e369509b011993f613170addca3097fc2db76f1daf WatchSource:0}: Error finding container 08d0099708d66788226d18e369509b011993f613170addca3097fc2db76f1daf: Status 404 returned error can't find the container with id 08d0099708d66788226d18e369509b011993f613170addca3097fc2db76f1daf Nov 02 01:45:33.665874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.665841 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.665923 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666000 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.665989 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.666025 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666077 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.666061 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.666143 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666231 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.666216 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666325 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.666312 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.666384 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.666371 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data\") pod \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\" (UID: \"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552\") " Nov 02 01:45:33.670307 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.670274 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.670539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.670518 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.671069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.671043 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.672844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.672814 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.673483 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.673425 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.673849 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.673670 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.674433 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.674404 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:33.674837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.674724 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.675892 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.675871 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999" pod "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" (UID: "1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552"). InnerVolumeSpecName "kube-api-access-vh999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.679493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.679162 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:45:33.688706 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.688639 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:45:33.768170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.767864 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.768170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.767929 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.768170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.767963 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.768170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.768014 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.768170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.768048 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.768170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.768086 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.769261 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.769645 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.775359 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.776099 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.776320 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.775796 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777423 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777509 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777572 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777655 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777742 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777798 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777842 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777880 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777934 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs\") pod \"fe252814-5eec-4e00-9e30-013c83dbd6d0\" (UID: \"fe252814-5eec-4e00-9e30-013c83dbd6d0\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.777973 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.778218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.778043 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token\") pod \"c9560f4a-a1ac-49c2-9829-728c386c6260\" (UID: \"c9560f4a-a1ac-49c2-9829-728c386c6260\") " Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779675 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779703 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779713 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779722 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779731 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779739 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779747 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9560f4a-a1ac-49c2-9829-728c386c6260-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779756 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779767 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe252814-5eec-4e00-9e30-013c83dbd6d0-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779780 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh999\" (UniqueName: \"kubernetes.io/projected/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-kube-api-access-vh999\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779789 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779797 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779808 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.779816 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.780768 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.780993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.780899 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.781983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.781233 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.784496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.782837 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.784496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.783093 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.784496 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.783367 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.784814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.784784 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "kube-api-access-zhmw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.785032 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.785010 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "kube-api-access-64wzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.785446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.785414 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:33.785446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.785438 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:33.786848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.786820 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.788192 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.788163 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy" pod "c9560f4a-a1ac-49c2-9829-728c386c6260" (UID: "c9560f4a-a1ac-49c2-9829-728c386c6260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.788476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.788440 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token" pod "fe252814-5eec-4e00-9e30-013c83dbd6d0" (UID: "fe252814-5eec-4e00-9e30-013c83dbd6d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882310 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882351 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882367 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882383 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882397 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882409 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882422 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882437 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9560f4a-a1ac-49c2-9829-728c386c6260-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882451 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64wzr\" (UniqueName: \"kubernetes.io/projected/fe252814-5eec-4e00-9e30-013c83dbd6d0-kube-api-access-64wzr\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882465 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe252814-5eec-4e00-9e30-013c83dbd6d0-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882480 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhmw9\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-kube-api-access-zhmw9\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882494 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9560f4a-a1ac-49c2-9829-728c386c6260-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.882532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.882507 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe252814-5eec-4e00-9e30-013c83dbd6d0-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.912493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.912463 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:45:33.983544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983513 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.983765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983561 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.983765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983679 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.983765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983712 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.983765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983761 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.984052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983795 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.984052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.983880 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.984052 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.984010 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.984252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.984052 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket\") pod \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\" (UID: \"6f6519d4-abe0-4ce9-bc38-38993a7ad791\") " Nov 02 01:45:33.985168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.985139 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.985432 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.985409 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.985639 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.985600 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.985792 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.985771 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.986181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.986155 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.986503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.986471 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:33.987029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.987002 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "kube-api-access-xv78s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.988174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.988140 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.988269 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:33.988190 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy" pod "6f6519d4-abe0-4ce9-bc38-38993a7ad791" (UID: "6f6519d4-abe0-4ce9-bc38-38993a7ad791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:34.087391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087328 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087363 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087379 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087392 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv78s\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-kube-api-access-xv78s\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087406 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087418 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087430 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6519d4-abe0-4ce9-bc38-38993a7ad791-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087441 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.087803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.087452 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6519d4-abe0-4ce9-bc38-38993a7ad791-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.438213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.438079 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" event={"ID":"1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552","Type":"ContainerDied","Data":"64fef0833413b0e09bb0e035217b316dd00f03aafdc43b8560f60fe018d86602"} Nov 02 01:45:34.438213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.438192 4552 scope.go:122] "RemoveContainer" containerID="d11056c596f14bd73e52c222653bdf3d503748b4fee140b0ea3c31d2d48a37af" Nov 02 01:45:34.438429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.438087 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz" Nov 02 01:45:34.440837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.440811 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" Nov 02 01:45:34.441326 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.441299 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89" event={"ID":"c9560f4a-a1ac-49c2-9829-728c386c6260","Type":"ContainerDied","Data":"5de347349e2429124c9f10dd8024be0c088e30248bdf700f3046b570d38ab9b8"} Nov 02 01:45:34.443515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.443488 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" event={"ID":"b7add8ae-4a07-44e6-ae69-be43510e6dce","Type":"ContainerStarted","Data":"73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038"} Nov 02 01:45:34.443694 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.443526 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" event={"ID":"b7add8ae-4a07-44e6-ae69-be43510e6dce","Type":"ContainerStarted","Data":"08d0099708d66788226d18e369509b011993f613170addca3097fc2db76f1daf"} Nov 02 01:45:34.446213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.446186 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" event={"ID":"fe252814-5eec-4e00-9e30-013c83dbd6d0","Type":"ContainerDied","Data":"fd3a0532c6f5d009e254923afd2da44624289569738d4eb0af46ed056ba7ba2c"} Nov 02 01:45:34.446310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.446271 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5" Nov 02 01:45:34.449685 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.449659 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" Nov 02 01:45:34.449807 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.449705 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh" event={"ID":"6f6519d4-abe0-4ce9-bc38-38993a7ad791","Type":"ContainerDied","Data":"3a535aca9219d6fb0f7c81ea0e12eb91ea9581a7f812b0d2b43f5164c21c7107"} Nov 02 01:45:34.454053 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.454039 4552 scope.go:122] "RemoveContainer" containerID="8443997b7217ef0f4c0572281d24cbffc76dc202277414cb5e12450755bef2ee" Nov 02 01:45:34.466359 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.466329 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89"] Nov 02 01:45:34.468490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.468469 4552 scope.go:122] "RemoveContainer" containerID="ad63ca75c4ed7106d03ed3e847e0276da8797b198c2ee936ce59167688163a87" Nov 02 01:45:34.472387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.472363 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-jdm89"] Nov 02 01:45:34.481565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.481544 4552 scope.go:122] "RemoveContainer" containerID="d43fe3baddcd96c4d2b6907d29c23790e5007c79a8d27a4dea3b8e204fcdc8af" Nov 02 01:45:34.486560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.486530 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz"] Nov 02 01:45:34.490814 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.490790 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-psjkz"] Nov 02 01:45:34.507991 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.507948 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" podStartSLOduration=2.5079349669999997 podStartE2EDuration="2.507934967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:34.503853594 +0000 UTC m=+1834.609642509" watchObservedRunningTime="2026-11-02 01:45:34.507934967 +0000 UTC m=+1834.613723881" Nov 02 01:45:34.521260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.521224 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh"] Nov 02 01:45:34.525042 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.525017 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-sv2zh"] Nov 02 01:45:34.535100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.535072 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5"] Nov 02 01:45:34.548061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:34.547808 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-bhvs5"] Nov 02 01:45:35.228356 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:35.228323 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:36.049009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:36.048971 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" path="/var/lib/kubelet/pods/1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552/volumes" Nov 02 01:45:36.050147 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:36.050089 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" path="/var/lib/kubelet/pods/6f6519d4-abe0-4ce9-bc38-38993a7ad791/volumes" Nov 02 01:45:36.050691 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:36.050673 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" path="/var/lib/kubelet/pods/c9560f4a-a1ac-49c2-9829-728c386c6260/volumes" Nov 02 01:45:36.051427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:36.051408 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" path="/var/lib/kubelet/pods/fe252814-5eec-4e00-9e30-013c83dbd6d0/volumes" Nov 02 01:45:36.138451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:36.138418 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:45:42.169288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.167177 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt"] Nov 02 01:45:42.169288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.168500 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" podUID="1d32b0e1-25be-41dc-a8b2-e86476d4c70d" containerName="maas-api" containerID="cri-o://c2896528a6d0a19a8a6c9a816f2e8c3d21914a671678378b23e2ce99c6ec4557" gracePeriod=30 Nov 02 01:45:42.528067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.526043 4552 generic.go:356] "Generic (PLEG): container finished" podID="1d32b0e1-25be-41dc-a8b2-e86476d4c70d" containerID="c2896528a6d0a19a8a6c9a816f2e8c3d21914a671678378b23e2ce99c6ec4557" exitCode=0 Nov 02 01:45:42.528067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.526314 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" event={"ID":"1d32b0e1-25be-41dc-a8b2-e86476d4c70d","Type":"ContainerDied","Data":"c2896528a6d0a19a8a6c9a816f2e8c3d21914a671678378b23e2ce99c6ec4557"} Nov 02 01:45:42.702095 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.702069 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:45:42.840439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.840379 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca\") pod \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " Nov 02 01:45:42.841940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.840902 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs\") pod \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " Nov 02 01:45:42.841940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.841003 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls\") pod \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " Nov 02 01:45:42.841940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.840997 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca" pod "1d32b0e1-25be-41dc-a8b2-e86476d4c70d" (UID: "1d32b0e1-25be-41dc-a8b2-e86476d4c70d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:42.841940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.841032 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms\" (UniqueName: \"kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms\") pod \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\" (UID: \"1d32b0e1-25be-41dc-a8b2-e86476d4c70d\") " Nov 02 01:45:42.841940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.841817 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:42.846314 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.846262 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls" pod "1d32b0e1-25be-41dc-a8b2-e86476d4c70d" (UID: "1d32b0e1-25be-41dc-a8b2-e86476d4c70d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:42.850131 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.850075 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs" pod "1d32b0e1-25be-41dc-a8b2-e86476d4c70d" (UID: "1d32b0e1-25be-41dc-a8b2-e86476d4c70d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:42.851101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.850345 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms" pod "1d32b0e1-25be-41dc-a8b2-e86476d4c70d" (UID: "1d32b0e1-25be-41dc-a8b2-e86476d4c70d"). InnerVolumeSpecName "kube-api-access-b67ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:42.943441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.943399 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:42.943441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.943434 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:42.943441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:42.943445 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b67ms\" (UniqueName: \"kubernetes.io/projected/1d32b0e1-25be-41dc-a8b2-e86476d4c70d-kube-api-access-b67ms\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:43.227977 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.227942 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:43.247097 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.247061 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:43.539816 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.539714 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" Nov 02 01:45:43.539816 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.539765 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt" event={"ID":"1d32b0e1-25be-41dc-a8b2-e86476d4c70d","Type":"ContainerDied","Data":"42c90e0f4796b353cb6000a04778aa22c0dc907a0834db999bd675efe96f6a89"} Nov 02 01:45:43.540060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.539854 4552 scope.go:122] "RemoveContainer" containerID="c2896528a6d0a19a8a6c9a816f2e8c3d21914a671678378b23e2ce99c6ec4557" Nov 02 01:45:43.556929 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.556900 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:45:43.571319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.571292 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt"] Nov 02 01:45:43.581795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:43.581742 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf1e55-5b4758687b-5qgzt"] Nov 02 01:45:44.052599 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.052559 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d32b0e1-25be-41dc-a8b2-e86476d4c70d" path="/var/lib/kubelet/pods/1d32b0e1-25be-41dc-a8b2-e86476d4c70d/volumes" Nov 02 01:45:44.074049 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.074005 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx"] Nov 02 01:45:44.074300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.074271 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" podUID="7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" containerName="maas-api" containerID="cri-o://0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1" gracePeriod=30 Nov 02 01:45:44.506389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.506362 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:45:44.556194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.556142 4552 generic.go:356] "Generic (PLEG): container finished" podID="7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" containerID="0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1" exitCode=0 Nov 02 01:45:44.556412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.556238 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" event={"ID":"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26","Type":"ContainerDied","Data":"0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1"} Nov 02 01:45:44.556412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.556304 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" event={"ID":"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26","Type":"ContainerDied","Data":"f49cb9bb301ddde19d75a424aa754bef5329a20889ae890bc468e5589cc9ce86"} Nov 02 01:45:44.556412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.556333 4552 scope.go:122] "RemoveContainer" containerID="0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1" Nov 02 01:45:44.556547 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.556458 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx" Nov 02 01:45:44.570950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.570926 4552 scope.go:122] "RemoveContainer" containerID="0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1" Nov 02 01:45:44.571617 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:45:44.571440 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1\": container with ID starting with 0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1 not found: ID does not exist" containerID="0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1" Nov 02 01:45:44.571617 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.571477 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1"} err="failed to get container status \"0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1\": rpc error: code = NotFound desc = could not find container \"0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1\": container with ID starting with 0ce9509be45bb280efd1b7e76e844c42ea2a6e795f6fa3455c24daf87e954fa1 not found: ID does not exist" Nov 02 01:45:44.595498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.595469 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn\" (UniqueName: \"kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn\") pod \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " Nov 02 01:45:44.595498 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.595517 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca\") pod \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " Nov 02 01:45:44.595762 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.595552 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls\") pod \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " Nov 02 01:45:44.595762 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.595602 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs\") pod \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\" (UID: \"7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26\") " Nov 02 01:45:44.595989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.595962 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca" pod "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" (UID: "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:44.596508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.596431 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:44.598102 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.598075 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs" pod "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" (UID: "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:44.598273 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.598187 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn" pod "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" (UID: "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26"). InnerVolumeSpecName "kube-api-access-8spbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:44.598322 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.598298 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls" pod "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" (UID: "7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:44.698098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.698061 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8spbn\" (UniqueName: \"kubernetes.io/projected/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-kube-api-access-8spbn\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:44.698098 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.698096 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:44.698386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.698112 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:44.884830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.884799 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx"] Nov 02 01:45:44.888707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:44.888679 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bbb385-fc8fd5ddf-hr8vx"] Nov 02 01:45:46.021457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.021420 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g"] Nov 02 01:45:46.022513 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022490 4552 cpu_manager.go:414] "containerMap: removing container" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" Nov 02 01:45:46.022513 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022515 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022534 4552 cpu_manager.go:414] "containerMap: removing container" podUID="1d32b0e1-25be-41dc-a8b2-e86476d4c70d" containerName="maas-api" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022543 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d32b0e1-25be-41dc-a8b2-e86476d4c70d" containerName="maas-api" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022561 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022569 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022610 4552 cpu_manager.go:414] "containerMap: removing container" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022617 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022642 4552 cpu_manager.go:414] "containerMap: removing container" podUID="7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" containerName="maas-api" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022651 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" containerName="maas-api" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022669 4552 cpu_manager.go:414] "containerMap: removing container" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" Nov 02 01:45:46.022674 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022675 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" Nov 02 01:45:46.023021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022845 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4d4da0-8e13-4dec-82ab-1fd2dbb0d552" containerName="istio-proxy" Nov 02 01:45:46.023021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022864 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d32b0e1-25be-41dc-a8b2-e86476d4c70d" containerName="maas-api" Nov 02 01:45:46.023021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022875 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9560f4a-a1ac-49c2-9829-728c386c6260" containerName="istio-proxy" Nov 02 01:45:46.023021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022890 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe252814-5eec-4e00-9e30-013c83dbd6d0" containerName="istio-proxy" Nov 02 01:45:46.023021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022902 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" containerName="maas-api" Nov 02 01:45:46.023021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.022913 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6519d4-abe0-4ce9-bc38-38993a7ad791" containerName="istio-proxy" Nov 02 01:45:46.028832 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.028808 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.035211 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.035181 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g"] Nov 02 01:45:46.035451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.035427 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0f32d6\"/\"default-dockercfg-sbsmg\"" Nov 02 01:45:46.035634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.035617 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0f32d6\"/\"auth-test-model-0f32d6-kserve-self-signed-certs\"" Nov 02 01:45:46.036586 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.036564 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0f32d6\"/\"openshift-service-ca.crt\"" Nov 02 01:45:46.036698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.036626 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0f32d6\"/\"kube-root-ca.crt\"" Nov 02 01:45:46.053039 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.053003 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26" path="/var/lib/kubelet/pods/7b758427-1b0b-4b7d-9fa5-d2bd7f5d5c26/volumes" Nov 02 01:45:46.220155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.219995 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.220444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.220195 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.220444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.220361 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdwr\" (UniqueName: \"kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.220444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.220412 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.220653 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.220475 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.220653 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.220530 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322304 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdwr\" (UniqueName: \"kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322345 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322393 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322380 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322407 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322509 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322687 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322560 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322849 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322803 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322928 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.322990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.322953 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.325182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.325157 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.325429 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.325408 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.334694 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.334673 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdwr\" (UniqueName: \"kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr\") pod \"auth-test-model-0f32d6-kserve-6787c999bf-srj2g\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.353626 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.353604 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:46.682747 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:46.682715 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g"] Nov 02 01:45:46.683530 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:45:46.683496 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ccb7ca1_4b23_4e0f_88bc_9dce000ae5ca.slice/crio-e20c6708f71726ec65add1aed74c609c1428a08d774bf9283ef8d96f71e9f9db WatchSource:0}: Error finding container e20c6708f71726ec65add1aed74c609c1428a08d774bf9283ef8d96f71e9f9db: Status 404 returned error can't find the container with id e20c6708f71726ec65add1aed74c609c1428a08d774bf9283ef8d96f71e9f9db Nov 02 01:45:47.497502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.497464 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl"] Nov 02 01:45:47.502158 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.502088 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.509609 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.509564 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c8d4a0-openshift-default-dockercfg-qcqst\"" Nov 02 01:45:47.513064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.513033 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl"] Nov 02 01:45:47.585736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.585691 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" event={"ID":"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca","Type":"ContainerStarted","Data":"663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263"} Nov 02 01:45:47.585915 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.585760 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" event={"ID":"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca","Type":"ContainerStarted","Data":"e20c6708f71726ec65add1aed74c609c1428a08d774bf9283ef8d96f71e9f9db"} Nov 02 01:45:47.609064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.608568 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" podStartSLOduration=1.608547529 podStartE2EDuration="1.608547529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:47.60667982 +0000 UTC m=+1847.712468735" watchObservedRunningTime="2026-11-02 01:45:47.608547529 +0000 UTC m=+1847.714336437" Nov 02 01:45:47.641692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.641638 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.641911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.641735 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rj9\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.641911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.641772 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.641911 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.641809 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.642179 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.641952 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.642179 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.642012 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.642179 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.642078 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.642348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.642196 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.642348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.642222 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744066 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744178 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744241 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744333 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744356 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744478 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744530 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rj9\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744564 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.744601 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.745566 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.746468 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.747111 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.747462 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.749479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.748762 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.752328 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.752300 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.758427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.758370 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rj9\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.763165 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.761612 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.765676 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.765653 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:47.824765 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:47.824726 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:48.133882 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:45:48.133848 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1281cd4_30ee_45b9_9c22_932403b8eec2.slice/crio-d80f64fc7eaf31b7196410f134ba0e1874ee80c30b0b183439a4934f801a3a78 WatchSource:0}: Error finding container d80f64fc7eaf31b7196410f134ba0e1874ee80c30b0b183439a4934f801a3a78: Status 404 returned error can't find the container with id d80f64fc7eaf31b7196410f134ba0e1874ee80c30b0b183439a4934f801a3a78 Nov 02 01:45:48.134474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.134448 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl"] Nov 02 01:45:48.136809 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.136773 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:45:48.136919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.136853 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:45:48.136919 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.136901 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:45:48.353822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.353784 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:48.599210 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.599173 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" event={"ID":"d1281cd4-30ee-45b9-9c22-932403b8eec2","Type":"ContainerStarted","Data":"76b29ebea7b579d23294e3ccb87fbd29cbc2e5a3af816317efedfa07ee442b53"} Nov 02 01:45:48.599763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.599727 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" event={"ID":"d1281cd4-30ee-45b9-9c22-932403b8eec2","Type":"ContainerStarted","Data":"d80f64fc7eaf31b7196410f134ba0e1874ee80c30b0b183439a4934f801a3a78"} Nov 02 01:45:48.624903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.624848 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podStartSLOduration=1.6248336 podStartE2EDuration="1.6248336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:48.621652928 +0000 UTC m=+1848.727441844" watchObservedRunningTime="2026-11-02 01:45:48.6248336 +0000 UTC m=+1848.730622513" Nov 02 01:45:48.825713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:48.825673 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:49.827745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:49.827706 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 01:45:49.828291 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:49.827772 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 01:45:50.812995 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:50.812715 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf"] Nov 02 01:45:50.813725 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:50.813662 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" containerID="cri-o://0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a" gracePeriod=30 Nov 02 01:45:50.826028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:50.825070 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 01:45:50.826028 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:50.825146 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 01:45:51.825047 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:51.825016 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 01:45:51.825465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:51.825065 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 01:45:52.825901 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:52.825866 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 01:45:52.826336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:52.825917 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 01:45:52.934354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:52.934219 4552 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:52.934354 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:52.934283 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:53.825521 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:53.825486 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 01:45:53.825709 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:53.825537 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 01:45:54.829590 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:54.829561 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:54.830068 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:54.829913 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:54.830741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:54.830723 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:45:56.118788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.118761 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:45:56.162353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162319 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162397 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162472 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162534 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162510 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162543 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162637 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162663 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162716 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.162844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162711 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:56.162844 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.162824 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs\") pod \"835e5898-3c65-480d-8b5c-917f255de079\" (UID: \"835e5898-3c65-480d-8b5c-917f255de079\") " Nov 02 01:45:56.163022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.163000 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.163261 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.163240 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.163603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.163588 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.163659 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.163611 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.163659 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.163626 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835e5898-3c65-480d-8b5c-917f255de079-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.164276 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.164069 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.164416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.164306 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.171610 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.171586 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.179875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.179842 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:56.179983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.179866 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.182088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.182049 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz" pod "835e5898-3c65-480d-8b5c-917f255de079" (UID: "835e5898-3c65-480d-8b5c-917f255de079"). InnerVolumeSpecName "kube-api-access-hmmlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.265020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.264989 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.265020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.265017 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.265020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.265026 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835e5898-3c65-480d-8b5c-917f255de079-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.265020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.265036 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmlz\" (UniqueName: \"kubernetes.io/projected/835e5898-3c65-480d-8b5c-917f255de079-kube-api-access-hmmlz\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.265415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.265050 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.265415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.265061 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835e5898-3c65-480d-8b5c-917f255de079-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.358195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.357767 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:56.375947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.375916 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:56.699389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.699294 4552 generic.go:356] "Generic (PLEG): container finished" podID="835e5898-3c65-480d-8b5c-917f255de079" containerID="0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a" exitCode=0 Nov 02 01:45:56.699389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.699358 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" Nov 02 01:45:56.699389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.699378 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" event={"ID":"835e5898-3c65-480d-8b5c-917f255de079","Type":"ContainerDied","Data":"0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a"} Nov 02 01:45:56.699681 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.699433 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf" event={"ID":"835e5898-3c65-480d-8b5c-917f255de079","Type":"ContainerDied","Data":"6583f9f4b0a7defd6fefef9c81415ddde648e4196259df76b40270f8b62cb29c"} Nov 02 01:45:56.699681 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.699462 4552 scope.go:122] "RemoveContainer" containerID="0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a" Nov 02 01:45:56.712493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.712472 4552 scope.go:122] "RemoveContainer" containerID="0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a" Nov 02 01:45:56.712774 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:45:56.712753 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a\": container with ID starting with 0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a not found: ID does not exist" containerID="0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a" Nov 02 01:45:56.712857 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.712785 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a"} err="failed to get container status \"0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a\": rpc error: code = NotFound desc = could not find container \"0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a\": container with ID starting with 0a180acddec6114b5e02dd3419e9aeed88b47ed494d6f37752f4e3c03b157b5a not found: ID does not exist" Nov 02 01:45:56.715079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.715056 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:45:56.725905 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.725883 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf"] Nov 02 01:45:56.730340 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:56.730319 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-58rtf"] Nov 02 01:45:57.758149 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:45:57.758059 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:45:58.068195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:45:58.059433 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835e5898-3c65-480d-8b5c-917f255de079" path="/var/lib/kubelet/pods/835e5898-3c65-480d-8b5c-917f255de079/volumes" Nov 02 01:46:04.530456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:04.530419 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k"] Nov 02 01:46:04.530917 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:04.530677 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" podUID="9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" containerName="maas-api" containerID="cri-o://21ba71f4ab39f4b641c9cf63d789a36dc4bef6d78caa059a9e1dfe7dc0a6a236" gracePeriod=30 Nov 02 01:46:04.927474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:04.927445 4552 generic.go:356] "Generic (PLEG): container finished" podID="9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" containerID="21ba71f4ab39f4b641c9cf63d789a36dc4bef6d78caa059a9e1dfe7dc0a6a236" exitCode=0 Nov 02 01:46:04.927713 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:04.927682 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" event={"ID":"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033","Type":"ContainerDied","Data":"21ba71f4ab39f4b641c9cf63d789a36dc4bef6d78caa059a9e1dfe7dc0a6a236"} Nov 02 01:46:04.950209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:04.950184 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:46:05.042760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.042732 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca\") pod \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " Nov 02 01:46:05.042968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.042805 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9\" (UniqueName: \"kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9\") pod \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " Nov 02 01:46:05.042968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.042910 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs\") pod \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " Nov 02 01:46:05.043096 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.043011 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls\") pod \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\" (UID: \"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033\") " Nov 02 01:46:05.043224 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.043195 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca" pod "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" (UID: "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:05.043466 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.043448 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:05.045291 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.045267 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9" pod "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" (UID: "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033"). InnerVolumeSpecName "kube-api-access-h7zl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:05.045414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.045397 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls" pod "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" (UID: "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:05.045501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.045486 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs" pod "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" (UID: "9926cc0c-382d-4e2f-b1c0-3d03cc9f5033"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:05.169152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.168305 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zl9\" (UniqueName: \"kubernetes.io/projected/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-kube-api-access-h7zl9\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:05.169152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.168341 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:05.169152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.168356 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:05.936992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.936953 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" Nov 02 01:46:05.937548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.936992 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k" event={"ID":"9926cc0c-382d-4e2f-b1c0-3d03cc9f5033","Type":"ContainerDied","Data":"d85348c9329561b5835a30dce98cbac37c5940384082a6f33e1c1e9e3e593b3d"} Nov 02 01:46:05.937548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.937068 4552 scope.go:122] "RemoveContainer" containerID="21ba71f4ab39f4b641c9cf63d789a36dc4bef6d78caa059a9e1dfe7dc0a6a236" Nov 02 01:46:05.965707 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.965672 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k"] Nov 02 01:46:05.973298 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:05.973274 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-00f1da-55cfc8d778-7d26k"] Nov 02 01:46:06.047425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:06.047390 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" path="/var/lib/kubelet/pods/9926cc0c-382d-4e2f-b1c0-3d03cc9f5033/volumes" Nov 02 01:46:12.496705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:12.496662 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc"] Nov 02 01:46:12.497579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:12.496941 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" containerID="cri-o://60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5" gracePeriod=30 Nov 02 01:46:17.799293 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.799267 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:46:17.884454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884417 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.884657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884582 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.884657 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884634 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.884788 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884664 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.884993 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884962 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:17.885084 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884998 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:17.885084 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.884772 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.885260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885102 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.885260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885170 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.885260 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885211 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.885416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885281 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy\") pod \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\" (UID: \"f7f40f3a-964f-4d39-bde6-e75bc4e136c2\") " Nov 02 01:46:17.885416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885364 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:17.885515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885446 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:17.885881 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885857 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:17.885984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885932 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.885984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885963 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.885984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885979 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.886184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.885993 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.888309 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.888282 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "kube-api-access-tj2wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:17.888566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.888534 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:17.888999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.888969 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:17.896604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.896579 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy" pod "f7f40f3a-964f-4d39-bde6-e75bc4e136c2" (UID: "f7f40f3a-964f-4d39-bde6-e75bc4e136c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:17.988476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.988368 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.988476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.988406 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.988476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.988421 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.988476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.988434 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.988476 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:17.988449 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj2wc\" (UniqueName: \"kubernetes.io/projected/f7f40f3a-964f-4d39-bde6-e75bc4e136c2-kube-api-access-tj2wc\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:18.056525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.056431 4552 generic.go:356] "Generic (PLEG): container finished" podID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerID="60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5" exitCode=0 Nov 02 01:46:18.056525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.056475 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" Nov 02 01:46:18.056525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.056513 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" event={"ID":"f7f40f3a-964f-4d39-bde6-e75bc4e136c2","Type":"ContainerDied","Data":"60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5"} Nov 02 01:46:18.056830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.056576 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc" event={"ID":"f7f40f3a-964f-4d39-bde6-e75bc4e136c2","Type":"ContainerDied","Data":"6630f9eb0e407772904a33a4b8f98a45fd94adf6fb178bf9e074714e3319d330"} Nov 02 01:46:18.056830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.056606 4552 scope.go:122] "RemoveContainer" containerID="60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5" Nov 02 01:46:18.069468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.069448 4552 scope.go:122] "RemoveContainer" containerID="60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5" Nov 02 01:46:18.069788 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:18.069769 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5\": container with ID starting with 60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5 not found: ID does not exist" containerID="60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5" Nov 02 01:46:18.069858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.069795 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5"} err="failed to get container status \"60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5\": rpc error: code = NotFound desc = could not find container \"60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5\": container with ID starting with 60db45f1c1a1a0f2c7028c8c7b270c9f9af35ff5864eed4b02937d788cee3be5 not found: ID does not exist" Nov 02 01:46:18.089463 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.089414 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc"] Nov 02 01:46:18.094252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:18.094227 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-hh2rc"] Nov 02 01:46:20.048442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:20.048406 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" path="/var/lib/kubelet/pods/f7f40f3a-964f-4d39-bde6-e75bc4e136c2/volumes" Nov 02 01:46:23.306746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:23.306712 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d"] Nov 02 01:46:23.514904 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:23.514859 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8"] Nov 02 01:46:23.533961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:23.533931 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp"] Nov 02 01:46:27.768156 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:27.768081 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:46:28.515234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:28.515163 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" podUID="0187afc3-ac44-4424-b308-c332cdaac014" containerName="payload-processing" containerID="cri-o://a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a" gracePeriod=25 Nov 02 01:46:28.539932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:28.539544 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerName="payload-pre-processing" containerID="cri-o://5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090" gracePeriod=25 Nov 02 01:46:28.728877 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:28.728846 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc"] Nov 02 01:46:28.739695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:28.739662 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv"] Nov 02 01:46:29.526813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:29.526784 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:46:33.729723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:33.729678 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerName="payload-processing" containerID="cri-o://20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73" gracePeriod=25 Nov 02 01:46:33.745026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:33.744980 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" podUID="1b58629a-2db8-4e59-a467-f9b597164194" containerName="payload-pre-processing" containerID="cri-o://edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e" gracePeriod=25 Nov 02 01:46:36.100992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:36.100949 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" podUID="0187afc3-ac44-4424-b308-c332cdaac014" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:36.101529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:36.101070 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" podUID="1b58629a-2db8-4e59-a467-f9b597164194" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:36.101529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:36.101480 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:36.101736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:36.101702 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:38.331080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.331042 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" containerID="cri-o://bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845" gracePeriod=45 Nov 02 01:46:38.631522 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.631499 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:46:38.655845 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.655811 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir\") pod \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " Nov 02 01:46:38.656441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.656324 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir" pod "deb78ec3-fde7-42fd-ae62-4d03e15f2190" (UID: "deb78ec3-fde7-42fd-ae62-4d03e15f2190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.656585 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.656411 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv\" (UniqueName: \"kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv\") pod \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " Nov 02 01:46:38.656659 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.656575 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache\") pod \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " Nov 02 01:46:38.656659 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.656630 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs\" (UniqueName: \"kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs\") pod \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " Nov 02 01:46:38.657022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.656996 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache" pod "deb78ec3-fde7-42fd-ae62-4d03e15f2190" (UID: "deb78ec3-fde7-42fd-ae62-4d03e15f2190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.657155 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.657099 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm\") pod \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " Nov 02 01:46:38.657268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.657249 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home\") pod \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\" (UID: \"deb78ec3-fde7-42fd-ae62-4d03e15f2190\") " Nov 02 01:46:38.659427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.657717 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home" pod "deb78ec3-fde7-42fd-ae62-4d03e15f2190" (UID: "deb78ec3-fde7-42fd-ae62-4d03e15f2190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.659427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.658429 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.659427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.658448 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.659427 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.658461 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.666617 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.664439 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm" pod "deb78ec3-fde7-42fd-ae62-4d03e15f2190" (UID: "deb78ec3-fde7-42fd-ae62-4d03e15f2190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.666617 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.664608 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv" pod "deb78ec3-fde7-42fd-ae62-4d03e15f2190" (UID: "deb78ec3-fde7-42fd-ae62-4d03e15f2190"). InnerVolumeSpecName "kube-api-access-r78pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:38.666617 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.665210 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs" pod "deb78ec3-fde7-42fd-ae62-4d03e15f2190" (UID: "deb78ec3-fde7-42fd-ae62-4d03e15f2190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:38.760290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.760260 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78pv\" (UniqueName: \"kubernetes.io/projected/deb78ec3-fde7-42fd-ae62-4d03e15f2190-kube-api-access-r78pv\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.760290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.760287 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb78ec3-fde7-42fd-ae62-4d03e15f2190-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.760290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:38.760297 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb78ec3-fde7-42fd-ae62-4d03e15f2190-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:39.245990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.245953 4552 generic.go:356] "Generic (PLEG): container finished" podID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerID="bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845" exitCode=0 Nov 02 01:46:39.246233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.246020 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" Nov 02 01:46:39.246233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.246041 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" event={"ID":"deb78ec3-fde7-42fd-ae62-4d03e15f2190","Type":"ContainerDied","Data":"bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845"} Nov 02 01:46:39.246233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.246093 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d" event={"ID":"deb78ec3-fde7-42fd-ae62-4d03e15f2190","Type":"ContainerDied","Data":"9210cf2b9fa39739915d70951097c16f3c00321297ebe847507b2e7e6c18e273"} Nov 02 01:46:39.246233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.246141 4552 scope.go:122] "RemoveContainer" containerID="bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845" Nov 02 01:46:39.260634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.260590 4552 scope.go:122] "RemoveContainer" containerID="bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845" Nov 02 01:46:39.261014 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:39.260988 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845\": container with ID starting with bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845 not found: ID does not exist" containerID="bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845" Nov 02 01:46:39.261192 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.261025 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845"} err="failed to get container status \"bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845\": rpc error: code = NotFound desc = could not find container \"bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845\": container with ID starting with bc0b6695eb72491917e7822998e12a37b3f23146fc83c52a41e8db06489d6845 not found: ID does not exist" Nov 02 01:46:39.274884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.274854 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d"] Nov 02 01:46:39.281831 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:39.281803 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-w6-b9d816-kserve-67f8c9d989-whh8d"] Nov 02 01:46:40.059304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:40.059259 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" path="/var/lib/kubelet/pods/deb78ec3-fde7-42fd-ae62-4d03e15f2190/volumes" Nov 02 01:46:41.648445 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:41.648407 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg"] Nov 02 01:46:41.649353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:41.649290 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" podUID="cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" containerName="maas-api" containerID="cri-o://2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4" gracePeriod=30 Nov 02 01:46:42.043449 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.043416 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:46:42.121454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.121417 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs\") pod \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " Nov 02 01:46:42.121624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.121565 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls\") pod \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " Nov 02 01:46:42.121624 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.121618 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r\" (UniqueName: \"kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r\") pod \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " Nov 02 01:46:42.121813 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.121797 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca\") pod \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\" (UID: \"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d\") " Nov 02 01:46:42.122184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.122147 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca" pod "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" (UID: "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:42.122504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.122400 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:42.123874 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.123854 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r" pod "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" (UID: "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d"). InnerVolumeSpecName "kube-api-access-4s82r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:42.124305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.124287 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls" pod "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" (UID: "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:42.124396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.124318 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs" pod "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" (UID: "cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:42.223885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.223851 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s82r\" (UniqueName: \"kubernetes.io/projected/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-kube-api-access-4s82r\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:42.223885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.223886 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:42.223885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.223901 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:42.272592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.272508 4552 generic.go:356] "Generic (PLEG): container finished" podID="cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" containerID="2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4" exitCode=0 Nov 02 01:46:42.272592 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.272554 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" Nov 02 01:46:42.272768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.272582 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" event={"ID":"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d","Type":"ContainerDied","Data":"2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4"} Nov 02 01:46:42.272768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.272643 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg" event={"ID":"cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d","Type":"ContainerDied","Data":"f979dac47d27e7c7741dfe6777d601fd74d4bd01c51e370e8cd88cf9e10d70b7"} Nov 02 01:46:42.272768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.272669 4552 scope.go:122] "RemoveContainer" containerID="2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4" Nov 02 01:46:42.285493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.285475 4552 scope.go:122] "RemoveContainer" containerID="2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4" Nov 02 01:46:42.285800 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:42.285781 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4\": container with ID starting with 2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4 not found: ID does not exist" containerID="2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4" Nov 02 01:46:42.285864 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.285809 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4"} err="failed to get container status \"2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4\": rpc error: code = NotFound desc = could not find container \"2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4\": container with ID starting with 2da587efa8731c15e6ddc11471bc45189892b22270ac291d67fe8e405a5130e4 not found: ID does not exist" Nov 02 01:46:42.296989 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.296921 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg"] Nov 02 01:46:42.301749 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:42.301720 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b9d816-b5f8bcf7c-7blxg"] Nov 02 01:46:44.046882 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:44.046848 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" path="/var/lib/kubelet/pods/cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d/volumes" Nov 02 01:46:45.509409 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:45.509376 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5"] Nov 02 01:46:45.509921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:45.509630 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" podUID="642275e4-de7e-4784-92a6-05f1bc69095b" containerName="maas-api" containerID="cri-o://492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04" gracePeriod=30 Nov 02 01:46:45.877303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:45.877267 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:46:46.002773 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.002735 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs\") pod \"642275e4-de7e-4784-92a6-05f1bc69095b\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " Nov 02 01:46:46.002955 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.002796 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca\") pod \"642275e4-de7e-4784-92a6-05f1bc69095b\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " Nov 02 01:46:46.002955 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.002892 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr\" (UniqueName: \"kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr\") pod \"642275e4-de7e-4784-92a6-05f1bc69095b\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " Nov 02 01:46:46.003110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.002968 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls\") pod \"642275e4-de7e-4784-92a6-05f1bc69095b\" (UID: \"642275e4-de7e-4784-92a6-05f1bc69095b\") " Nov 02 01:46:46.003328 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.003294 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca" pod "642275e4-de7e-4784-92a6-05f1bc69095b" (UID: "642275e4-de7e-4784-92a6-05f1bc69095b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:46.003544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.003519 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642275e4-de7e-4784-92a6-05f1bc69095b-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:46.005297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.005270 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs" pod "642275e4-de7e-4784-92a6-05f1bc69095b" (UID: "642275e4-de7e-4784-92a6-05f1bc69095b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:46.005297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.005288 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls" pod "642275e4-de7e-4784-92a6-05f1bc69095b" (UID: "642275e4-de7e-4784-92a6-05f1bc69095b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:46.005403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.005391 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr" pod "642275e4-de7e-4784-92a6-05f1bc69095b" (UID: "642275e4-de7e-4784-92a6-05f1bc69095b"). InnerVolumeSpecName "kube-api-access-59msr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:46.101424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.101237 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" podUID="1b58629a-2db8-4e59-a467-f9b597164194" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:46.101424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.101277 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" podUID="0187afc3-ac44-4424-b308-c332cdaac014" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:46.101623 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.101602 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:46.101796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.101770 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:46:46.104659 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.104633 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:46.104659 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.104658 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642275e4-de7e-4784-92a6-05f1bc69095b-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:46.104828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.104668 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59msr\" (UniqueName: \"kubernetes.io/projected/642275e4-de7e-4784-92a6-05f1bc69095b-kube-api-access-59msr\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:46.309867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.309828 4552 generic.go:356] "Generic (PLEG): container finished" podID="642275e4-de7e-4784-92a6-05f1bc69095b" containerID="492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04" exitCode=0 Nov 02 01:46:46.310079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.309891 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" Nov 02 01:46:46.310079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.309915 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" event={"ID":"642275e4-de7e-4784-92a6-05f1bc69095b","Type":"ContainerDied","Data":"492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04"} Nov 02 01:46:46.310079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.309982 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5" event={"ID":"642275e4-de7e-4784-92a6-05f1bc69095b","Type":"ContainerDied","Data":"64e6d8eebd0727728d5416d730ab3b2123e469b4ede74960df42d876a01977ff"} Nov 02 01:46:46.310079 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.310012 4552 scope.go:122] "RemoveContainer" containerID="492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04" Nov 02 01:46:46.330304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.330274 4552 scope.go:122] "RemoveContainer" containerID="492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04" Nov 02 01:46:46.330690 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:46.330660 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04\": container with ID starting with 492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04 not found: ID does not exist" containerID="492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04" Nov 02 01:46:46.330910 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.330874 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04"} err="failed to get container status \"492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04\": rpc error: code = NotFound desc = could not find container \"492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04\": container with ID starting with 492bb0a51a715394064fb4dbdbd4f1d38f7bd7a4bbac5908313da2ea88a59b04 not found: ID does not exist" Nov 02 01:46:46.332901 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.332878 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5"] Nov 02 01:46:46.337064 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:46.337041 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5687bc-7db4784dfc-f8mv5"] Nov 02 01:46:48.044793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:48.044759 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642275e4-de7e-4784-92a6-05f1bc69095b" path="/var/lib/kubelet/pods/642275e4-de7e-4784-92a6-05f1bc69095b/volumes" Nov 02 01:46:48.930451 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:48.930427 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:46:48.979751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:48.979722 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:46:49.052577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.052477 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume\") pod \"0187afc3-ac44-4424-b308-c332cdaac014\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " Nov 02 01:46:49.052577 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.052529 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn\" (UniqueName: \"kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn\") pod \"0187afc3-ac44-4424-b308-c332cdaac014\" (UID: \"0187afc3-ac44-4424-b308-c332cdaac014\") " Nov 02 01:46:49.053040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.052904 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume" pod "0187afc3-ac44-4424-b308-c332cdaac014" (UID: "0187afc3-ac44-4424-b308-c332cdaac014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:49.053088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.053060 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0187afc3-ac44-4424-b308-c332cdaac014-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:49.054924 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.054899 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn" pod "0187afc3-ac44-4424-b308-c332cdaac014" (UID: "0187afc3-ac44-4424-b308-c332cdaac014"). InnerVolumeSpecName "kube-api-access-l57pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:49.153903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.153864 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86\" (UniqueName: \"kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86\") pod \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " Nov 02 01:46:49.154153 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.154107 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume\") pod \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\" (UID: \"f21d48af-7b46-4cf8-83f9-7d2e0d74d061\") " Nov 02 01:46:49.154868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.154438 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume" pod "f21d48af-7b46-4cf8-83f9-7d2e0d74d061" (UID: "f21d48af-7b46-4cf8-83f9-7d2e0d74d061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:49.154868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.154731 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:49.154868 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.154753 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l57pn\" (UniqueName: \"kubernetes.io/projected/0187afc3-ac44-4424-b308-c332cdaac014-kube-api-access-l57pn\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:49.156289 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.156266 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86" pod "f21d48af-7b46-4cf8-83f9-7d2e0d74d061" (UID: "f21d48af-7b46-4cf8-83f9-7d2e0d74d061"). InnerVolumeSpecName "kube-api-access-26n86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:49.256295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.256261 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26n86\" (UniqueName: \"kubernetes.io/projected/f21d48af-7b46-4cf8-83f9-7d2e0d74d061-kube-api-access-26n86\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:49.342643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.342546 4552 generic.go:356] "Generic (PLEG): container finished" podID="0187afc3-ac44-4424-b308-c332cdaac014" containerID="a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a" exitCode=0 Nov 02 01:46:49.342643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.342609 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" Nov 02 01:46:49.342869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.342746 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" event={"ID":"0187afc3-ac44-4424-b308-c332cdaac014","Type":"ContainerDied","Data":"a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a"} Nov 02 01:46:49.342869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.342795 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8" event={"ID":"0187afc3-ac44-4424-b308-c332cdaac014","Type":"ContainerDied","Data":"8b22f7cdb149d4e5a714398c577b43d36708a91a89294408c022ef9124f4c503"} Nov 02 01:46:49.342869 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.342823 4552 scope.go:122] "RemoveContainer" containerID="a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a" Nov 02 01:46:49.347775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.346204 4552 generic.go:356] "Generic (PLEG): container finished" podID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerID="5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090" exitCode=0 Nov 02 01:46:49.347775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.346286 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" event={"ID":"f21d48af-7b46-4cf8-83f9-7d2e0d74d061","Type":"ContainerDied","Data":"5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090"} Nov 02 01:46:49.347775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.346337 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" event={"ID":"f21d48af-7b46-4cf8-83f9-7d2e0d74d061","Type":"ContainerDied","Data":"28c0558d522f7ac037bdf8a633b23baf5b7ef1651a01267befeb2f7f90f6c5d8"} Nov 02 01:46:49.347775 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.346411 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp" Nov 02 01:46:49.365583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.365524 4552 scope.go:122] "RemoveContainer" containerID="a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a" Nov 02 01:46:49.366017 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:49.365982 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a\": container with ID starting with a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a not found: ID does not exist" containerID="a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a" Nov 02 01:46:49.366184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.366030 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a"} err="failed to get container status \"a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a\": rpc error: code = NotFound desc = could not find container \"a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a\": container with ID starting with a082a71f230caf17079c72ccfabe5ae74f8d5b3b9b36691c6c834f4543d2697a not found: ID does not exist" Nov 02 01:46:49.366184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.366055 4552 scope.go:122] "RemoveContainer" containerID="5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090" Nov 02 01:46:49.391353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.391321 4552 scope.go:122] "RemoveContainer" containerID="5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090" Nov 02 01:46:49.391765 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:49.391729 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090\": container with ID starting with 5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090 not found: ID does not exist" containerID="5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090" Nov 02 01:46:49.391836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.391773 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090"} err="failed to get container status \"5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090\": rpc error: code = NotFound desc = could not find container \"5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090\": container with ID starting with 5dcc3cf34348f2a0977f96b1c356c903d2234b191dc973066d9650861691a090 not found: ID does not exist" Nov 02 01:46:49.392549 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.392503 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp"] Nov 02 01:46:49.398074 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.398053 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5687bc-748dbd68f9-vrjgp"] Nov 02 01:46:49.404489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.404470 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8"] Nov 02 01:46:49.408871 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.408851 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5687bc-d9b894d95-jzng8"] Nov 02 01:46:49.505434 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.505376 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5"] Nov 02 01:46:49.505647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.505615 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" containerID="cri-o://ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054" gracePeriod=30 Nov 02 01:46:49.517566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.513872 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6"] Nov 02 01:46:49.517879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:49.517813 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" containerID="cri-o://81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a" gracePeriod=30 Nov 02 01:46:50.046836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:50.046806 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0187afc3-ac44-4424-b308-c332cdaac014" path="/var/lib/kubelet/pods/0187afc3-ac44-4424-b308-c332cdaac014/volumes" Nov 02 01:46:50.047234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:50.047219 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" path="/var/lib/kubelet/pods/f21d48af-7b46-4cf8-83f9-7d2e0d74d061/volumes" Nov 02 01:46:53.089743 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:53.089705 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb"] Nov 02 01:46:53.090351 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:53.089937 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" containerID="cri-o://eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1" gracePeriod=30 Nov 02 01:46:54.060265 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.060230 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:46:54.110421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.110388 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:46:54.228566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.228533 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9\" (UniqueName: \"kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9\") pod \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " Nov 02 01:46:54.228745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.228624 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume\") pod \"1b58629a-2db8-4e59-a467-f9b597164194\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " Nov 02 01:46:54.228745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.228686 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf\" (UniqueName: \"kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf\") pod \"1b58629a-2db8-4e59-a467-f9b597164194\" (UID: \"1b58629a-2db8-4e59-a467-f9b597164194\") " Nov 02 01:46:54.228745 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.228734 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume\") pod \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\" (UID: \"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4\") " Nov 02 01:46:54.229143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.229096 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume" pod "1b58629a-2db8-4e59-a467-f9b597164194" (UID: "1b58629a-2db8-4e59-a467-f9b597164194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:54.229310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.229152 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume" pod "c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" (UID: "c9f8a4bf-fc12-45ed-bb85-245fbf05bde4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:54.229379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.229308 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b58629a-2db8-4e59-a467-f9b597164194-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:54.229379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.229328 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:54.230958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.230938 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf" pod "1b58629a-2db8-4e59-a467-f9b597164194" (UID: "1b58629a-2db8-4e59-a467-f9b597164194"). InnerVolumeSpecName "kube-api-access-s49xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:54.231067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.230953 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9" pod "c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" (UID: "c9f8a4bf-fc12-45ed-bb85-245fbf05bde4"). InnerVolumeSpecName "kube-api-access-dltl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:54.330870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.330794 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltl9\" (UniqueName: \"kubernetes.io/projected/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4-kube-api-access-dltl9\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:54.330870 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.330822 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s49xf\" (UniqueName: \"kubernetes.io/projected/1b58629a-2db8-4e59-a467-f9b597164194-kube-api-access-s49xf\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:54.392295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.392259 4552 generic.go:356] "Generic (PLEG): container finished" podID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerID="20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73" exitCode=0 Nov 02 01:46:54.392482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.392323 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" Nov 02 01:46:54.392482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.392342 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" event={"ID":"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4","Type":"ContainerDied","Data":"20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73"} Nov 02 01:46:54.392482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.392430 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc" event={"ID":"c9f8a4bf-fc12-45ed-bb85-245fbf05bde4","Type":"ContainerDied","Data":"60ef273828f4cc84917a5b3c4484d332ecefaa9e562193a4587a0e198c79ddd1"} Nov 02 01:46:54.392482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.392463 4552 scope.go:122] "RemoveContainer" containerID="20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73" Nov 02 01:46:54.393865 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.393840 4552 generic.go:356] "Generic (PLEG): container finished" podID="1b58629a-2db8-4e59-a467-f9b597164194" containerID="edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e" exitCode=0 Nov 02 01:46:54.394003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.393875 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" event={"ID":"1b58629a-2db8-4e59-a467-f9b597164194","Type":"ContainerDied","Data":"edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e"} Nov 02 01:46:54.394003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.393895 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" Nov 02 01:46:54.394003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.393903 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv" event={"ID":"1b58629a-2db8-4e59-a467-f9b597164194","Type":"ContainerDied","Data":"a1b2ae250ca1a26b860a4e2c044ef3aaa6776f5150681ed11c7ccf7a54aefa5c"} Nov 02 01:46:54.406046 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.406024 4552 scope.go:122] "RemoveContainer" containerID="20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73" Nov 02 01:46:54.406443 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:54.406416 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73\": container with ID starting with 20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73 not found: ID does not exist" containerID="20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73" Nov 02 01:46:54.406586 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.406456 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73"} err="failed to get container status \"20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73\": rpc error: code = NotFound desc = could not find container \"20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73\": container with ID starting with 20b5898ae21581cd7a5f2fef2581fb2fe2c9fdc5ed45273419e4403bc7a2df73 not found: ID does not exist" Nov 02 01:46:54.406586 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.406482 4552 scope.go:122] "RemoveContainer" containerID="edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e" Nov 02 01:46:54.418297 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.418270 4552 scope.go:122] "RemoveContainer" containerID="edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e" Nov 02 01:46:54.418557 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:54.418541 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e\": container with ID starting with edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e not found: ID does not exist" containerID="edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e" Nov 02 01:46:54.418607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.418566 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e"} err="failed to get container status \"edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e\": rpc error: code = NotFound desc = could not find container \"edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e\": container with ID starting with edcb09b99d57d7d65fdfebff75e381c82cda8516b2ed3014aea9204edf26655e not found: ID does not exist" Nov 02 01:46:54.425520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.425498 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc"] Nov 02 01:46:54.432748 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.432726 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9d816-c9877fbb7-kzlzc"] Nov 02 01:46:54.445213 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.445185 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv"] Nov 02 01:46:54.446958 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.446939 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9d816-b8dcf964c-k42rv"] Nov 02 01:46:54.874969 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.874899 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963071 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963162 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963209 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963289 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963434 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963474 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963525 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963582 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.963629 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data\") pod \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\" (UID: \"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f\") " Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.964788 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.964844 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.965014 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.965764 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:54.968087 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.966297 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.969836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.969290 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:54.989194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.984522 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:54.989194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.986717 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "kube-api-access-r97gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:54.989194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.988601 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:46:54.989194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:54.988750 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy" pod "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" (UID: "fe69b2e4-f5ff-48b6-bb41-5b5de507f74f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:55.071643 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.071615 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.071935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.071682 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.071935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.071717 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.071935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.071798 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.071935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.071862 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.072378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072000 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.072378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072034 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.072378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072100 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.072378 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072247 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data\") pod \"e281771e-1091-4348-b47f-a7c5152c9cdc\" (UID: \"e281771e-1091-4348-b47f-a7c5152c9cdc\") " Nov 02 01:46:55.072930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072901 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.072930 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072927 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072942 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97gw\" (UniqueName: \"kubernetes.io/projected/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-kube-api-access-r97gw\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072955 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072969 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072982 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.072996 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.073010 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.073201 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.073025 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.077802 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.077775 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:55.077990 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.077961 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:55.081044 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.081018 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:55.081883 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.081023 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:55.084878 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.081821 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:55.112031 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.111941 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "kube-api-access-ptkjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:55.112743 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.112708 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:55.114544 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.114516 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:55.114626 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.114572 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token" pod "e281771e-1091-4348-b47f-a7c5152c9cdc" (UID: "e281771e-1091-4348-b47f-a7c5152c9cdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:55.174249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174136 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174189 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174204 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174218 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptkjb\" (UniqueName: \"kubernetes.io/projected/e281771e-1091-4348-b47f-a7c5152c9cdc-kube-api-access-ptkjb\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174235 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e281771e-1091-4348-b47f-a7c5152c9cdc-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174248 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174264 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174279 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e281771e-1091-4348-b47f-a7c5152c9cdc-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.174716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.174293 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e281771e-1091-4348-b47f-a7c5152c9cdc-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.407444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.407406 4552 generic.go:356] "Generic (PLEG): container finished" podID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerID="81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a" exitCode=0 Nov 02 01:46:55.407444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.407434 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" event={"ID":"e281771e-1091-4348-b47f-a7c5152c9cdc","Type":"ContainerDied","Data":"81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a"} Nov 02 01:46:55.407712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.407486 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" Nov 02 01:46:55.407712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.407533 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6" event={"ID":"e281771e-1091-4348-b47f-a7c5152c9cdc","Type":"ContainerDied","Data":"f4679f3f29822c03b36b4a116a6695f25ed89cfe7ecacdeb6f92e532a4339d2a"} Nov 02 01:46:55.407712 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.407580 4552 scope.go:122] "RemoveContainer" containerID="81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a" Nov 02 01:46:55.410110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.409852 4552 generic.go:356] "Generic (PLEG): container finished" podID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerID="ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054" exitCode=0 Nov 02 01:46:55.410110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.409895 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" Nov 02 01:46:55.410110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.410042 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" event={"ID":"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f","Type":"ContainerDied","Data":"ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054"} Nov 02 01:46:55.410110 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.410075 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5" event={"ID":"fe69b2e4-f5ff-48b6-bb41-5b5de507f74f","Type":"ContainerDied","Data":"62ae71e1bcdd1e29b3e6ca225c72a6af0ec35fee9821c2cbad5d81731c77369c"} Nov 02 01:46:55.423629 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.423608 4552 scope.go:122] "RemoveContainer" containerID="81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a" Nov 02 01:46:55.424165 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:55.424143 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a\": container with ID starting with 81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a not found: ID does not exist" containerID="81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a" Nov 02 01:46:55.424757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.424177 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a"} err="failed to get container status \"81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a\": rpc error: code = NotFound desc = could not find container \"81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a\": container with ID starting with 81b51e6f89e99c957d5f6f8c32c4d16138975a8eff5d40eccb4dcd05daeed48a not found: ID does not exist" Nov 02 01:46:55.424757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.424200 4552 scope.go:122] "RemoveContainer" containerID="ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054" Nov 02 01:46:55.435266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.435248 4552 scope.go:122] "RemoveContainer" containerID="ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054" Nov 02 01:46:55.435557 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:55.435539 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054\": container with ID starting with ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054 not found: ID does not exist" containerID="ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054" Nov 02 01:46:55.435618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.435566 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054"} err="failed to get container status \"ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054\": rpc error: code = NotFound desc = could not find container \"ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054\": container with ID starting with ab9c97ee0ac34d1c2e2a861ec4c4c921405993fbef066722fc6f9fc769c3c054 not found: ID does not exist" Nov 02 01:46:55.441708 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.441658 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6"] Nov 02 01:46:55.447428 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.447408 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-9ptj6"] Nov 02 01:46:55.460374 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.460353 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5"] Nov 02 01:46:55.468108 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:55.468083 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-dclw5"] Nov 02 01:46:56.048278 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:56.048201 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b58629a-2db8-4e59-a467-f9b597164194" path="/var/lib/kubelet/pods/1b58629a-2db8-4e59-a467-f9b597164194/volumes" Nov 02 01:46:56.048867 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:56.048847 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" path="/var/lib/kubelet/pods/c9f8a4bf-fc12-45ed-bb85-245fbf05bde4/volumes" Nov 02 01:46:56.049480 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:56.049460 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" path="/var/lib/kubelet/pods/e281771e-1091-4348-b47f-a7c5152c9cdc/volumes" Nov 02 01:46:56.050489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:56.050468 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" path="/var/lib/kubelet/pods/fe69b2e4-f5ff-48b6-bb41-5b5de507f74f/volumes" Nov 02 01:46:57.751560 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:57.751518 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:46:58.387822 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.387794 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:46:58.414746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.414720 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.414906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.414805 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.414906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.414859 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415025 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.414925 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415025 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.414999 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415052 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415151 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415094 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.415278 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415100 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:58.415278 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415112 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415331 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415347 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.415399 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415388 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs\") pod \"a8ee346c-6716-419e-984f-016735e0c703\" (UID: \"a8ee346c-6716-419e-984f-016735e0c703\") " Nov 02 01:46:58.415567 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415389 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.415862 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.415834 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.416249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.416227 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ee346c-6716-419e-984f-016735e0c703-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.416346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.416257 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.416346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.416274 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.416346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.416306 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.416346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.416320 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.417353 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.417331 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:58.417447 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.417376 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "kube-api-access-5nbpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:58.417515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.417443 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.417608 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.417592 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo" pod "a8ee346c-6716-419e-984f-016735e0c703" (UID: "a8ee346c-6716-419e-984f-016735e0c703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:58.436431 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.436401 4552 generic.go:356] "Generic (PLEG): container finished" podID="a8ee346c-6716-419e-984f-016735e0c703" containerID="eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1" exitCode=0 Nov 02 01:46:58.436525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.436443 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" Nov 02 01:46:58.436525 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.436473 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" event={"ID":"a8ee346c-6716-419e-984f-016735e0c703","Type":"ContainerDied","Data":"eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1"} Nov 02 01:46:58.436600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.436524 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb" event={"ID":"a8ee346c-6716-419e-984f-016735e0c703","Type":"ContainerDied","Data":"bc079f1cf30afa8df8180017a19751301cc4e830f74e547dc57011e5963c9d7f"} Nov 02 01:46:58.436600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.436546 4552 scope.go:122] "RemoveContainer" containerID="eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1" Nov 02 01:46:58.447426 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.447406 4552 scope.go:122] "RemoveContainer" containerID="eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1" Nov 02 01:46:58.447738 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:46:58.447710 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1\": container with ID starting with eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1 not found: ID does not exist" containerID="eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1" Nov 02 01:46:58.447803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.447750 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1"} err="failed to get container status \"eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1\": rpc error: code = NotFound desc = could not find container \"eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1\": container with ID starting with eed4701a9ba5126552b6020202203e53a1402d74d4829bb84ad5b0a26c5670d1 not found: ID does not exist" Nov 02 01:46:58.514335 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.514302 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb"] Nov 02 01:46:58.517542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.517516 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ee346c-6716-419e-984f-016735e0c703-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.517542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.517541 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ee346c-6716-419e-984f-016735e0c703-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.517703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.517552 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nbpt\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-kube-api-access-5nbpt\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.517703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.517561 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ee346c-6716-419e-984f-016735e0c703-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.526581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:46:58.526552 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-zmwgb"] Nov 02 01:47:00.047217 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:47:00.047187 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ee346c-6716-419e-984f-016735e0c703" path="/var/lib/kubelet/pods/a8ee346c-6716-419e-984f-016735e0c703/volumes" Nov 02 01:47:18.666665 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:47:18.666577 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:47:27.754273 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:47:27.754236 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:47:44.349638 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:47:44.349598 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:47:57.751207 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:47:57.751168 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:48:03.976944 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:03.976904 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:48:09.793386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:09.793355 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz"] Nov 02 01:48:10.749243 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:10.749204 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p"] Nov 02 01:48:22.599459 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:22.599424 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq"] Nov 02 01:48:22.599932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:22.599689 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" podUID="f7ab0028-163a-43e4-950d-69c3b2f8c4f9" containerName="maas-api" containerID="cri-o://67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8" gracePeriod=30 Nov 02 01:48:22.983018 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:22.982979 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:48:23.132985 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.132953 4552 generic.go:356] "Generic (PLEG): container finished" podID="f7ab0028-163a-43e4-950d-69c3b2f8c4f9" containerID="67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8" exitCode=0 Nov 02 01:48:23.133235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.133006 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" Nov 02 01:48:23.133319 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.133271 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" event={"ID":"f7ab0028-163a-43e4-950d-69c3b2f8c4f9","Type":"ContainerDied","Data":"67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8"} Nov 02 01:48:23.133373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.133316 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq" event={"ID":"f7ab0028-163a-43e4-950d-69c3b2f8c4f9","Type":"ContainerDied","Data":"3c63bb97af84f85884c298debe244749500a83899c2e0e8a44e2e3cae00cf26b"} Nov 02 01:48:23.133373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.133346 4552 scope.go:122] "RemoveContainer" containerID="67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8" Nov 02 01:48:23.148948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.148921 4552 scope.go:122] "RemoveContainer" containerID="67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8" Nov 02 01:48:23.149444 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:23.149393 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8\": container with ID starting with 67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8 not found: ID does not exist" containerID="67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8" Nov 02 01:48:23.149444 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.149432 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8"} err="failed to get container status \"67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8\": rpc error: code = NotFound desc = could not find container \"67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8\": container with ID starting with 67e1b7460af5a9295afd4ca7c7674329170c08adea5b281def00b0e3120ffde8 not found: ID does not exist" Nov 02 01:48:23.179667 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.179641 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca\") pod \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " Nov 02 01:48:23.179803 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.179683 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs\") pod \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " Nov 02 01:48:23.179885 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.179870 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls\") pod \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " Nov 02 01:48:23.179940 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.179907 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb\" (UniqueName: \"kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb\") pod \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\" (UID: \"f7ab0028-163a-43e4-950d-69c3b2f8c4f9\") " Nov 02 01:48:23.180091 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.180067 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca" pod "f7ab0028-163a-43e4-950d-69c3b2f8c4f9" (UID: "f7ab0028-163a-43e4-950d-69c3b2f8c4f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:23.180714 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.180552 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:23.182253 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.182230 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb" pod "f7ab0028-163a-43e4-950d-69c3b2f8c4f9" (UID: "f7ab0028-163a-43e4-950d-69c3b2f8c4f9"). InnerVolumeSpecName "kube-api-access-7nggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:23.182348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.182281 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs" pod "f7ab0028-163a-43e4-950d-69c3b2f8c4f9" (UID: "f7ab0028-163a-43e4-950d-69c3b2f8c4f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:23.182484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.182466 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls" pod "f7ab0028-163a-43e4-950d-69c3b2f8c4f9" (UID: "f7ab0028-163a-43e4-950d-69c3b2f8c4f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:23.282763 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.282728 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:23.282951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.282774 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:23.282951 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.282789 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nggb\" (UniqueName: \"kubernetes.io/projected/f7ab0028-163a-43e4-950d-69c3b2f8c4f9-kube-api-access-7nggb\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:23.464219 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.464186 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq"] Nov 02 01:48:23.479067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:23.479033 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-677312-56f9fd4df7-pv5nq"] Nov 02 01:48:24.046328 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:24.046286 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ab0028-163a-43e4-950d-69c3b2f8c4f9" path="/var/lib/kubelet/pods/f7ab0028-163a-43e4-950d-69c3b2f8c4f9/volumes" Nov 02 01:48:24.816044 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:24.816001 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" podUID="cbd9253b-82f8-4930-91c8-9b2ff0cabce2" containerName="main" containerID="cri-o://5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756" gracePeriod=45 Nov 02 01:48:25.092252 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.092222 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:48:25.105196 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105173 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache\") pod \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " Nov 02 01:48:25.105332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105211 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home\") pod \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " Nov 02 01:48:25.105332 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105269 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm\") pod \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " Nov 02 01:48:25.105460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105349 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir\") pod \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " Nov 02 01:48:25.105460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105373 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md\" (UniqueName: \"kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md\") pod \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " Nov 02 01:48:25.105460 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105393 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs\") pod \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\" (UID: \"cbd9253b-82f8-4930-91c8-9b2ff0cabce2\") " Nov 02 01:48:25.105618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105452 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache" pod "cbd9253b-82f8-4930-91c8-9b2ff0cabce2" (UID: "cbd9253b-82f8-4930-91c8-9b2ff0cabce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:25.105618 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105534 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home" pod "cbd9253b-82f8-4930-91c8-9b2ff0cabce2" (UID: "cbd9253b-82f8-4930-91c8-9b2ff0cabce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:25.105738 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.105639 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir" pod "cbd9253b-82f8-4930-91c8-9b2ff0cabce2" (UID: "cbd9253b-82f8-4930-91c8-9b2ff0cabce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:25.106603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.106499 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:25.106603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.106522 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:25.106603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.106535 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:25.108313 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.108282 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md" pod "cbd9253b-82f8-4930-91c8-9b2ff0cabce2" (UID: "cbd9253b-82f8-4930-91c8-9b2ff0cabce2"). InnerVolumeSpecName "kube-api-access-zw6md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:25.108439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.108357 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs" pod "cbd9253b-82f8-4930-91c8-9b2ff0cabce2" (UID: "cbd9253b-82f8-4930-91c8-9b2ff0cabce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:25.108439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.108353 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm" pod "cbd9253b-82f8-4930-91c8-9b2ff0cabce2" (UID: "cbd9253b-82f8-4930-91c8-9b2ff0cabce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:25.150729 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.150693 4552 generic.go:356] "Generic (PLEG): container finished" podID="cbd9253b-82f8-4930-91c8-9b2ff0cabce2" containerID="5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756" exitCode=0 Nov 02 01:48:25.150895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.150753 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" Nov 02 01:48:25.150895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.150752 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" event={"ID":"cbd9253b-82f8-4930-91c8-9b2ff0cabce2","Type":"ContainerDied","Data":"5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756"} Nov 02 01:48:25.150895 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.150889 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz" event={"ID":"cbd9253b-82f8-4930-91c8-9b2ff0cabce2","Type":"ContainerDied","Data":"5cae687d674954c5fb5de17af154b2d6cb966ec984c704ae153cdb7c9cf94121"} Nov 02 01:48:25.151057 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.150913 4552 scope.go:122] "RemoveContainer" containerID="5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756" Nov 02 01:48:25.161336 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.161319 4552 scope.go:122] "RemoveContainer" containerID="5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756" Nov 02 01:48:25.161619 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:25.161593 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756\": container with ID starting with 5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756 not found: ID does not exist" containerID="5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756" Nov 02 01:48:25.161730 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.161625 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756"} err="failed to get container status \"5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756\": rpc error: code = NotFound desc = could not find container \"5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756\": container with ID starting with 5d8c68b94f9ec6d1765da83314d1062e7839074df999c369b072fcf47c4fe756 not found: ID does not exist" Nov 02 01:48:25.177772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.177341 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz"] Nov 02 01:48:25.180927 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.180903 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-677312/auth-test-model-677312-kserve-ff97cd7b5-9x9fz"] Nov 02 01:48:25.207695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.207673 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:25.207695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.207696 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6md\" (UniqueName: \"kubernetes.io/projected/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-kube-api-access-zw6md\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:25.207839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.207706 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd9253b-82f8-4930-91c8-9b2ff0cabce2-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:25.770836 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:25.770799 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" podUID="1f0a3813-6b8c-4908-8cef-9ec2621e179b" containerName="main" containerID="cri-o://2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341" gracePeriod=45 Nov 02 01:48:26.046523 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.046494 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd9253b-82f8-4930-91c8-9b2ff0cabce2" path="/var/lib/kubelet/pods/cbd9253b-82f8-4930-91c8-9b2ff0cabce2/volumes" Nov 02 01:48:26.061698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.061675 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:48:26.119970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.119933 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home\") pod \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120013 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm\") pod \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120051 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir\") pod \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120184 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs\") pod \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120205 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home" pod "1f0a3813-6b8c-4908-8cef-9ec2621e179b" (UID: "1f0a3813-6b8c-4908-8cef-9ec2621e179b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120245 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w\" (UniqueName: \"kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w\") pod \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120337 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache\") pod \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\" (UID: \"1f0a3813-6b8c-4908-8cef-9ec2621e179b\") " Nov 02 01:48:26.120474 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120345 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir" pod "1f0a3813-6b8c-4908-8cef-9ec2621e179b" (UID: "1f0a3813-6b8c-4908-8cef-9ec2621e179b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:26.120938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120915 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:26.121013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.120943 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:26.121382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.121359 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache" pod "1f0a3813-6b8c-4908-8cef-9ec2621e179b" (UID: "1f0a3813-6b8c-4908-8cef-9ec2621e179b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:26.123380 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.123358 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs" pod "1f0a3813-6b8c-4908-8cef-9ec2621e179b" (UID: "1f0a3813-6b8c-4908-8cef-9ec2621e179b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:26.123503 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.123410 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w" pod "1f0a3813-6b8c-4908-8cef-9ec2621e179b" (UID: "1f0a3813-6b8c-4908-8cef-9ec2621e179b"). InnerVolumeSpecName "kube-api-access-49z8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:26.123768 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.123753 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm" pod "1f0a3813-6b8c-4908-8cef-9ec2621e179b" (UID: "1f0a3813-6b8c-4908-8cef-9ec2621e179b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:26.168232 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.168195 4552 generic.go:356] "Generic (PLEG): container finished" podID="1f0a3813-6b8c-4908-8cef-9ec2621e179b" containerID="2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341" exitCode=0 Nov 02 01:48:26.168414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.168269 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" event={"ID":"1f0a3813-6b8c-4908-8cef-9ec2621e179b","Type":"ContainerDied","Data":"2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341"} Nov 02 01:48:26.168414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.168305 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" event={"ID":"1f0a3813-6b8c-4908-8cef-9ec2621e179b","Type":"ContainerDied","Data":"e09258a0c946ca46c930c35be159727639499732519039ad85028e2d33f22de7"} Nov 02 01:48:26.168414 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.168333 4552 scope.go:122] "RemoveContainer" containerID="2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341" Nov 02 01:48:26.168581 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.168507 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p" Nov 02 01:48:26.183804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.183776 4552 scope.go:122] "RemoveContainer" containerID="2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341" Nov 02 01:48:26.184238 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:26.184215 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341\": container with ID starting with 2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341 not found: ID does not exist" containerID="2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341" Nov 02 01:48:26.184331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.184249 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341"} err="failed to get container status \"2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341\": rpc error: code = NotFound desc = could not find container \"2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341\": container with ID starting with 2d7b4229b5aa737358175a14bd9031cc08571b92ff08b65b4b597ee83437b341 not found: ID does not exist" Nov 02 01:48:26.197173 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.197149 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p"] Nov 02 01:48:26.200560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.200539 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e06f2/auth-test-model-0e06f2-kserve-659b494d9-g4d8p"] Nov 02 01:48:26.222628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.222601 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:26.222628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.222625 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f0a3813-6b8c-4908-8cef-9ec2621e179b-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:26.222757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.222635 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f0a3813-6b8c-4908-8cef-9ec2621e179b-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:26.222757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:26.222644 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49z8w\" (UniqueName: \"kubernetes.io/projected/1f0a3813-6b8c-4908-8cef-9ec2621e179b-kube-api-access-49z8w\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:27.754527 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:27.754489 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:48:28.051983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:28.051892 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0a3813-6b8c-4908-8cef-9ec2621e179b" path="/var/lib/kubelet/pods/1f0a3813-6b8c-4908-8cef-9ec2621e179b/volumes" Nov 02 01:48:29.205857 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:29.205823 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd"] Nov 02 01:48:32.879234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:32.879195 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8"] Nov 02 01:48:34.856566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:34.856521 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:48:38.030529 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:38.030496 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g"] Nov 02 01:48:40.304144 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304043 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t"] Nov 02 01:48:40.304847 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304826 4552 cpu_manager.go:414] "containerMap: removing container" podUID="0187afc3-ac44-4424-b308-c332cdaac014" containerName="payload-processing" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304849 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187afc3-ac44-4424-b308-c332cdaac014" containerName="payload-processing" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304868 4552 cpu_manager.go:414] "containerMap: removing container" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerName="payload-processing" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304875 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerName="payload-processing" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304901 4552 cpu_manager.go:414] "containerMap: removing container" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304908 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304919 4552 cpu_manager.go:414] "containerMap: removing container" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304926 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304939 4552 cpu_manager.go:414] "containerMap: removing container" podUID="9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" containerName="maas-api" Nov 02 01:48:40.304946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304947 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304957 4552 cpu_manager.go:414] "containerMap: removing container" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304964 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304981 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.304989 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305002 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerName="payload-pre-processing" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305010 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerName="payload-pre-processing" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305031 4552 cpu_manager.go:414] "containerMap: removing container" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305038 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305047 4552 cpu_manager.go:414] "containerMap: removing container" podUID="cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305053 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305068 4552 cpu_manager.go:414] "containerMap: removing container" podUID="1b58629a-2db8-4e59-a467-f9b597164194" containerName="payload-pre-processing" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305074 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b58629a-2db8-4e59-a467-f9b597164194" containerName="payload-pre-processing" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305088 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f7ab0028-163a-43e4-950d-69c3b2f8c4f9" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305095 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ab0028-163a-43e4-950d-69c3b2f8c4f9" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305111 4552 cpu_manager.go:414] "containerMap: removing container" podUID="642275e4-de7e-4784-92a6-05f1bc69095b" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305135 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642275e4-de7e-4784-92a6-05f1bc69095b" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305154 4552 cpu_manager.go:414] "containerMap: removing container" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305161 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305172 4552 cpu_manager.go:414] "containerMap: removing container" podUID="cbd9253b-82f8-4930-91c8-9b2ff0cabce2" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305178 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd9253b-82f8-4930-91c8-9b2ff0cabce2" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305190 4552 cpu_manager.go:414] "containerMap: removing container" podUID="1f0a3813-6b8c-4908-8cef-9ec2621e179b" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305197 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0a3813-6b8c-4908-8cef-9ec2621e179b" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305329 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ab0028-163a-43e4-950d-69c3b2f8c4f9" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305342 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="642275e4-de7e-4784-92a6-05f1bc69095b" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305353 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ee346c-6716-419e-984f-016735e0c703" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305363 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd9253b-82f8-4930-91c8-9b2ff0cabce2" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305371 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0a3813-6b8c-4908-8cef-9ec2621e179b" containerName="main" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305380 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187afc3-ac44-4424-b308-c332cdaac014" containerName="payload-processing" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305389 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f8a4bf-fc12-45ed-bb85-245fbf05bde4" containerName="payload-processing" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305399 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe69b2e4-f5ff-48b6-bb41-5b5de507f74f" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305407 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="e281771e-1091-4348-b47f-a7c5152c9cdc" containerName="istio-proxy" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305415 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="9926cc0c-382d-4e2f-b1c0-3d03cc9f5033" containerName="maas-api" Nov 02 01:48:40.305407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305423 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb78ec3-fde7-42fd-ae62-4d03e15f2190" containerName="main" Nov 02 01:48:40.307406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305432 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f40f3a-964f-4d39-bde6-e75bc4e136c2" containerName="istio-proxy" Nov 02 01:48:40.307406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305443 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21d48af-7b46-4cf8-83f9-7d2e0d74d061" containerName="payload-pre-processing" Nov 02 01:48:40.307406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305452 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="835e5898-3c65-480d-8b5c-917f255de079" containerName="istio-proxy" Nov 02 01:48:40.307406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305461 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc0aa9a-70cd-434b-b2a9-a502ff2ae86d" containerName="maas-api" Nov 02 01:48:40.307406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.305470 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b58629a-2db8-4e59-a467-f9b597164194" containerName="payload-pre-processing" Nov 02 01:48:40.311060 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.311040 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.317566 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.317539 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0f32d6-openshift-default-dockercfg-bh7jl\"" Nov 02 01:48:40.321303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.321280 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t"] Nov 02 01:48:40.415228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415193 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415249 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415357 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8b9\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415484 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415522 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415540 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415559 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415559 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415658 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.415716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.415701 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517282 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517340 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517378 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8b9\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517443 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517489 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517512 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517540 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517637 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517585 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517853 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517645 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.517938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517910 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.518005 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.517966 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.518104 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.518076 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.518256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.518172 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.518256 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.518219 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.520203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.520183 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.520421 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.520403 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.529303 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.529278 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.529396 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.529334 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8b9\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.627439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.627353 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:40.927994 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.927922 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t"] Nov 02 01:48:40.929574 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:48:40.929547 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a50d09_f5e3_4ddb_9955_248318b2082f.slice/crio-6b0b318c98724e309f9fec821ec8235c439ab2ec22853766f70e69ba8676903e WatchSource:0}: Error finding container 6b0b318c98724e309f9fec821ec8235c439ab2ec22853766f70e69ba8676903e: Status 404 returned error can't find the container with id 6b0b318c98724e309f9fec821ec8235c439ab2ec22853766f70e69ba8676903e Nov 02 01:48:40.932491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.932451 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:48:40.932602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.932538 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:48:40.932602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:40.932577 4552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 02 01:48:41.294669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:41.294627 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" event={"ID":"40a50d09-f5e3-4ddb-9955-248318b2082f","Type":"ContainerStarted","Data":"4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060"} Nov 02 01:48:41.294860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:41.294697 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" event={"ID":"40a50d09-f5e3-4ddb-9955-248318b2082f","Type":"ContainerStarted","Data":"6b0b318c98724e309f9fec821ec8235c439ab2ec22853766f70e69ba8676903e"} Nov 02 01:48:41.321162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:41.321097 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podStartSLOduration=1.321083069 podStartE2EDuration="1.321083069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:41.319835283 +0000 UTC m=+2021.425624197" watchObservedRunningTime="2026-11-02 01:48:41.321083069 +0000 UTC m=+2021.426871982" Nov 02 01:48:41.627504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:41.627428 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:42.630161 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:42.630100 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 01:48:42.630546 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:42.630184 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 01:48:43.628266 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:43.628199 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 01:48:43.628492 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:43.628290 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 01:48:44.227363 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.227292 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" podUID="b7add8ae-4a07-44e6-ae69-be43510e6dce" containerName="main" containerID="cri-o://73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038" gracePeriod=45 Nov 02 01:48:44.579508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.579482 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:48:44.614446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614417 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir\") pod \"b7add8ae-4a07-44e6-ae69-be43510e6dce\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " Nov 02 01:48:44.614615 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614493 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm\") pod \"b7add8ae-4a07-44e6-ae69-be43510e6dce\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " Nov 02 01:48:44.614673 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614610 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache\") pod \"b7add8ae-4a07-44e6-ae69-be43510e6dce\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " Nov 02 01:48:44.614754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614681 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home\") pod \"b7add8ae-4a07-44e6-ae69-be43510e6dce\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " Nov 02 01:48:44.614754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614719 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs\") pod \"b7add8ae-4a07-44e6-ae69-be43510e6dce\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " Nov 02 01:48:44.614754 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614729 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir" pod "b7add8ae-4a07-44e6-ae69-be43510e6dce" (UID: "b7add8ae-4a07-44e6-ae69-be43510e6dce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:44.614922 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614839 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j\" (UniqueName: \"kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j\") pod \"b7add8ae-4a07-44e6-ae69-be43510e6dce\" (UID: \"b7add8ae-4a07-44e6-ae69-be43510e6dce\") " Nov 02 01:48:44.614968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.614949 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache" pod "b7add8ae-4a07-44e6-ae69-be43510e6dce" (UID: "b7add8ae-4a07-44e6-ae69-be43510e6dce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:44.615398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.615074 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home" pod "b7add8ae-4a07-44e6-ae69-be43510e6dce" (UID: "b7add8ae-4a07-44e6-ae69-be43510e6dce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:44.615398 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.615392 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:44.615579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.615413 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:44.615579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.615425 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:44.617502 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.617476 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs" pod "b7add8ae-4a07-44e6-ae69-be43510e6dce" (UID: "b7add8ae-4a07-44e6-ae69-be43510e6dce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:44.617645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.617510 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j" pod "b7add8ae-4a07-44e6-ae69-be43510e6dce" (UID: "b7add8ae-4a07-44e6-ae69-be43510e6dce"). InnerVolumeSpecName "kube-api-access-rxt2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:44.617645 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.617638 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm" pod "b7add8ae-4a07-44e6-ae69-be43510e6dce" (UID: "b7add8ae-4a07-44e6-ae69-be43510e6dce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:44.628067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.628041 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 01:48:44.628216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.628091 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 01:48:44.716968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.716937 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7add8ae-4a07-44e6-ae69-be43510e6dce-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:44.716968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.716966 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxt2j\" (UniqueName: \"kubernetes.io/projected/b7add8ae-4a07-44e6-ae69-be43510e6dce-kube-api-access-rxt2j\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:44.716968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:44.716976 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7add8ae-4a07-44e6-ae69-be43510e6dce-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:45.326369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.326333 4552 generic.go:356] "Generic (PLEG): container finished" podID="b7add8ae-4a07-44e6-ae69-be43510e6dce" containerID="73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038" exitCode=0 Nov 02 01:48:45.326827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.326376 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" Nov 02 01:48:45.326827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.326415 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" event={"ID":"b7add8ae-4a07-44e6-ae69-be43510e6dce","Type":"ContainerDied","Data":"73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038"} Nov 02 01:48:45.326827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.326457 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" event={"ID":"b7add8ae-4a07-44e6-ae69-be43510e6dce","Type":"ContainerDied","Data":"08d0099708d66788226d18e369509b011993f613170addca3097fc2db76f1daf"} Nov 02 01:48:45.326827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.326475 4552 scope.go:122] "RemoveContainer" containerID="73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038" Nov 02 01:48:45.347820 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.347251 4552 scope.go:122] "RemoveContainer" containerID="73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038" Nov 02 01:48:45.347820 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:45.347585 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038\": container with ID starting with 73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038 not found: ID does not exist" containerID="73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038" Nov 02 01:48:45.347820 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.347618 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038"} err="failed to get container status \"73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038\": rpc error: code = NotFound desc = could not find container \"73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038\": container with ID starting with 73e864614ec6fd9512043ac7f73038357ddc6a7e3927291f772c14c01f0b8038 not found: ID does not exist" Nov 02 01:48:45.370178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.363401 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd"] Nov 02 01:48:45.373740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.373710 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd"] Nov 02 01:48:45.557143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.554204 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-3c4ea5/auth-test-model-3c4ea5-kserve-f5b5dfb9-9r2qd" podUID="b7add8ae-4a07-44e6-ae69-be43510e6dce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:45.628182 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.628076 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 01:48:45.628361 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:45.628178 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 01:48:46.069563 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.069524 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7add8ae-4a07-44e6-ae69-be43510e6dce" path="/var/lib/kubelet/pods/b7add8ae-4a07-44e6-ae69-be43510e6dce/volumes" Nov 02 01:48:46.190879 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.190833 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm"] Nov 02 01:48:46.191140 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.191097 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" podUID="fd03964c-70f0-4f6d-b23f-3e4e09650784" containerName="maas-api" containerID="cri-o://11b9b31de8b816531befd07ae2884c2baef00e9614763ebc73d955d1fe336a4e" gracePeriod=30 Nov 02 01:48:46.335865 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.335774 4552 generic.go:356] "Generic (PLEG): container finished" podID="fd03964c-70f0-4f6d-b23f-3e4e09650784" containerID="11b9b31de8b816531befd07ae2884c2baef00e9614763ebc73d955d1fe336a4e" exitCode=0 Nov 02 01:48:46.336465 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.335872 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" event={"ID":"fd03964c-70f0-4f6d-b23f-3e4e09650784","Type":"ContainerDied","Data":"11b9b31de8b816531befd07ae2884c2baef00e9614763ebc73d955d1fe336a4e"} Nov 02 01:48:46.627950 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.627919 4552 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 01:48:46.628112 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.627968 4552 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 01:48:46.633634 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.633611 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:48:46.654141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.649640 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca\") pod \"fd03964c-70f0-4f6d-b23f-3e4e09650784\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " Nov 02 01:48:46.654141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.649691 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs\") pod \"fd03964c-70f0-4f6d-b23f-3e4e09650784\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " Nov 02 01:48:46.654141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.649896 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls\") pod \"fd03964c-70f0-4f6d-b23f-3e4e09650784\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " Nov 02 01:48:46.654141 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.649991 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf\" (UniqueName: \"kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf\") pod \"fd03964c-70f0-4f6d-b23f-3e4e09650784\" (UID: \"fd03964c-70f0-4f6d-b23f-3e4e09650784\") " Nov 02 01:48:46.673138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.666611 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf" pod "fd03964c-70f0-4f6d-b23f-3e4e09650784" (UID: "fd03964c-70f0-4f6d-b23f-3e4e09650784"). InnerVolumeSpecName "kube-api-access-qp2rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:46.673138 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.668315 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca" pod "fd03964c-70f0-4f6d-b23f-3e4e09650784" (UID: "fd03964c-70f0-4f6d-b23f-3e4e09650784"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:46.681139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.680339 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs" pod "fd03964c-70f0-4f6d-b23f-3e4e09650784" (UID: "fd03964c-70f0-4f6d-b23f-3e4e09650784"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:46.704346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.699222 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls" pod "fd03964c-70f0-4f6d-b23f-3e4e09650784" (UID: "fd03964c-70f0-4f6d-b23f-3e4e09650784"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:46.761143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.753029 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:46.761143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.753067 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2rf\" (UniqueName: \"kubernetes.io/projected/fd03964c-70f0-4f6d-b23f-3e4e09650784-kube-api-access-qp2rf\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:46.761143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.753084 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd03964c-70f0-4f6d-b23f-3e4e09650784-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:46.761143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:46.753098 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd03964c-70f0-4f6d-b23f-3e4e09650784-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:47.353678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.353643 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" Nov 02 01:48:47.354287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.353704 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm" event={"ID":"fd03964c-70f0-4f6d-b23f-3e4e09650784","Type":"ContainerDied","Data":"7dba85fc27e57d8d89f494fe83507ccc20f7affebc13e2141025ff1c22391781"} Nov 02 01:48:47.354287 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.353808 4552 scope.go:122] "RemoveContainer" containerID="11b9b31de8b816531befd07ae2884c2baef00e9614763ebc73d955d1fe336a4e" Nov 02 01:48:47.382216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.382182 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm"] Nov 02 01:48:47.387539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.387509 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e06f2-c9fcb746c-z59cm"] Nov 02 01:48:47.633580 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.633507 4552 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:47.633805 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.633781 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:47.634488 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.634469 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:48:47.902073 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:47.901974 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" podUID="f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" containerName="main" containerID="cri-o://5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7" gracePeriod=45 Nov 02 01:48:48.047550 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.047513 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd03964c-70f0-4f6d-b23f-3e4e09650784" path="/var/lib/kubelet/pods/fd03964c-70f0-4f6d-b23f-3e4e09650784/volumes" Nov 02 01:48:48.189931 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.189905 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:48:48.272968 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.272931 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm\") pod \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " Nov 02 01:48:48.273162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.272984 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home\") pod \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " Nov 02 01:48:48.273162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273007 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir\") pod \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " Nov 02 01:48:48.273162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273043 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache\") pod \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " Nov 02 01:48:48.273162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273099 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79\" (UniqueName: \"kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79\") pod \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " Nov 02 01:48:48.273402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273239 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs\") pod \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\" (UID: \"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e\") " Nov 02 01:48:48.273402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273344 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir" pod "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" (UID: "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:48.273402 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273352 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home" pod "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" (UID: "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:48.273668 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273643 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache" pod "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" (UID: "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:48.273808 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273789 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:48.273873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273813 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:48.273873 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.273827 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:48.275897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.275838 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs" pod "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" (UID: "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:48.275897 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.275856 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm" pod "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" (UID: "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:48.276282 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.276251 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79" pod "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" (UID: "f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e"). InnerVolumeSpecName "kube-api-access-npm79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:48.361579 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.361540 4552 generic.go:356] "Generic (PLEG): container finished" podID="f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" containerID="5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7" exitCode=0 Nov 02 01:48:48.362030 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.361598 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" Nov 02 01:48:48.362030 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.361624 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" event={"ID":"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e","Type":"ContainerDied","Data":"5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7"} Nov 02 01:48:48.362030 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.361672 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8" event={"ID":"f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e","Type":"ContainerDied","Data":"7d489272b91a1f003998e3e4aebe7e70614f48f9a7eb189e8fce03f02d501c4c"} Nov 02 01:48:48.362030 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.361690 4552 scope.go:122] "RemoveContainer" containerID="5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7" Nov 02 01:48:48.376367 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.376336 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npm79\" (UniqueName: \"kubernetes.io/projected/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-kube-api-access-npm79\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:48.376501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.376371 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:48.376501 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.376387 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:48.394703 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.394672 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8"] Nov 02 01:48:48.395953 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.395929 4552 scope.go:122] "RemoveContainer" containerID="5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7" Nov 02 01:48:48.396385 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:48.396362 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7\": container with ID starting with 5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7 not found: ID does not exist" containerID="5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7" Nov 02 01:48:48.396479 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.396392 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7"} err="failed to get container status \"5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7\": rpc error: code = NotFound desc = could not find container \"5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7\": container with ID starting with 5e497999682d990e66d2e76315a896f07711ec2053fbbeb89d116f49a04a09b7 not found: ID does not exist" Nov 02 01:48:48.399026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:48.399005 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-eaec8c/auth-test-model-eaec8c-kserve-65cc4d5d85-nqld8"] Nov 02 01:48:50.049234 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:50.049199 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" path="/var/lib/kubelet/pods/f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e/volumes" Nov 02 01:48:53.056146 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.056075 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" podUID="9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" containerName="main" containerID="cri-o://663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263" gracePeriod=45 Nov 02 01:48:53.338893 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.338863 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:48:53.405426 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.405374 4552 generic.go:356] "Generic (PLEG): container finished" podID="9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" containerID="663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263" exitCode=0 Nov 02 01:48:53.405654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.405435 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" Nov 02 01:48:53.405654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.405451 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" event={"ID":"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca","Type":"ContainerDied","Data":"663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263"} Nov 02 01:48:53.405654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.405518 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g" event={"ID":"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca","Type":"ContainerDied","Data":"e20c6708f71726ec65add1aed74c609c1428a08d774bf9283ef8d96f71e9f9db"} Nov 02 01:48:53.405654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.405540 4552 scope.go:122] "RemoveContainer" containerID="663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263" Nov 02 01:48:53.418023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.417996 4552 scope.go:122] "RemoveContainer" containerID="663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263" Nov 02 01:48:53.418347 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:53.418325 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263\": container with ID starting with 663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263 not found: ID does not exist" containerID="663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263" Nov 02 01:48:53.418415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.418359 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263"} err="failed to get container status \"663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263\": rpc error: code = NotFound desc = could not find container \"663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263\": container with ID starting with 663333c4569576e05c6adcd2908af9dbb9b97a24f907d5318a19fe845c816263 not found: ID does not exist" Nov 02 01:48:53.458575 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.458540 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr\" (UniqueName: \"kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr\") pod \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " Nov 02 01:48:53.458722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.458688 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs\") pod \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " Nov 02 01:48:53.458808 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.458751 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache\") pod \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " Nov 02 01:48:53.458872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.458805 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir\") pod \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " Nov 02 01:48:53.458952 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.458936 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm\") pod \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " Nov 02 01:48:53.459022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.458984 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache" pod "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" (UID: "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:53.459130 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.459098 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home\") pod \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\" (UID: \"9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca\") " Nov 02 01:48:53.459212 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.459152 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir" pod "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" (UID: "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:53.459437 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.459415 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home" pod "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" (UID: "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:53.459619 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.459602 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:53.459682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.459626 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:53.459682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.459640 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:53.461181 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.461153 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs" pod "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" (UID: "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:53.461583 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.461560 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm" pod "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" (UID: "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:53.461692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.461583 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr" pod "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" (UID: "9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca"). InnerVolumeSpecName "kube-api-access-thdwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:53.563194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.562248 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:53.563194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.562291 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thdwr\" (UniqueName: \"kubernetes.io/projected/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-kube-api-access-thdwr\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:53.563194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.562307 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:53.733240 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.733207 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g"] Nov 02 01:48:53.737228 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:53.737207 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0f32d6/auth-test-model-0f32d6-kserve-6787c999bf-srj2g"] Nov 02 01:48:54.045392 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:54.045358 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" path="/var/lib/kubelet/pods/9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca/volumes" Nov 02 01:48:56.952876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:56.952833 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:48:57.783454 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:48:57.783396 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:48:59.561789 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:59.561728 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l"] Nov 02 01:48:59.572846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:48:59.572793 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5"] Nov 02 01:49:00.296935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:00.296901 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k"] Nov 02 01:49:04.605670 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:04.605632 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw"] Nov 02 01:49:04.606458 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:04.606302 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" podUID="2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" containerName="maas-api" containerID="cri-o://d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae" gracePeriod=30 Nov 02 01:49:04.939767 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:04.939735 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:49:05.071348 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.071311 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n\" (UniqueName: \"kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n\") pod \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " Nov 02 01:49:05.071519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.071362 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls\") pod \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " Nov 02 01:49:05.071519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.071431 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca\") pod \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " Nov 02 01:49:05.071606 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.071541 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs\") pod \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\" (UID: \"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5\") " Nov 02 01:49:05.071887 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.071851 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca" pod "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" (UID: "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:05.072050 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.072032 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.073731 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.073698 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs" pod "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" (UID: "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:05.073859 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.073737 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n" pod "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" (UID: "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5"). InnerVolumeSpecName "kube-api-access-h4k2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:05.073934 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.073925 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls" pod "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" (UID: "2d7d1de0-c5fa-40b6-9884-6e2cc11217e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:05.136292 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.136256 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc"] Nov 02 01:49:05.174834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.174708 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.174834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.174743 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4k2n\" (UniqueName: \"kubernetes.io/projected/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-kube-api-access-h4k2n\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.174834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.174759 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.297670 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.297625 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerName="payload-processing" containerID="cri-o://c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc" gracePeriod=25 Nov 02 01:49:05.520831 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.520798 4552 generic.go:356] "Generic (PLEG): container finished" podID="2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" containerID="d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae" exitCode=0 Nov 02 01:49:05.521037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.520842 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" Nov 02 01:49:05.521037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.520882 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" event={"ID":"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5","Type":"ContainerDied","Data":"d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae"} Nov 02 01:49:05.521037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.520938 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw" event={"ID":"2d7d1de0-c5fa-40b6-9884-6e2cc11217e5","Type":"ContainerDied","Data":"89c4b2e1cdac70d88e8f4485d335fd5c02c548914e991f55f3c2c123627afc0c"} Nov 02 01:49:05.521037 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.520960 4552 scope.go:122] "RemoveContainer" containerID="d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae" Nov 02 01:49:05.532034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.532017 4552 scope.go:122] "RemoveContainer" containerID="d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae" Nov 02 01:49:05.532333 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:05.532315 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae\": container with ID starting with d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae not found: ID does not exist" containerID="d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae" Nov 02 01:49:05.532423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.532340 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae"} err="failed to get container status \"d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae\": rpc error: code = NotFound desc = could not find container \"d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae\": container with ID starting with d793d269eb984c5f0f249540026c6fad0442c4245015d3ad845edf42dbda2bae not found: ID does not exist" Nov 02 01:49:05.547539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.547515 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw"] Nov 02 01:49:05.553306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:05.553287 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d4a0-687cd464ff-92xnw"] Nov 02 01:49:06.046101 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:06.046065 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" path="/var/lib/kubelet/pods/2d7d1de0-c5fa-40b6-9884-6e2cc11217e5/volumes" Nov 02 01:49:10.136600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:10.136533 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerName="payload-pre-processing" containerID="cri-o://bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887" gracePeriod=25 Nov 02 01:49:12.378543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:12.378503 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl"] Nov 02 01:49:12.379038 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:12.378748 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" containerID="cri-o://76b29ebea7b579d23294e3ccb87fbd29cbc2e5a3af816317efedfa07ee442b53" gracePeriod=30 Nov 02 01:49:12.885139 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:12.884942 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg"] Nov 02 01:49:12.885446 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:12.885266 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" podUID="0d357153-7b81-45fd-9a70-11e58828c720" containerName="maas-api" containerID="cri-o://af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e" gracePeriod=30 Nov 02 01:49:13.218739 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.218713 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:49:13.229394 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.229373 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs\") pod \"0d357153-7b81-45fd-9a70-11e58828c720\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " Nov 02 01:49:13.229521 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.229509 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2\" (UniqueName: \"kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2\") pod \"0d357153-7b81-45fd-9a70-11e58828c720\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " Nov 02 01:49:13.229561 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.229533 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca\") pod \"0d357153-7b81-45fd-9a70-11e58828c720\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " Nov 02 01:49:13.229603 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.229558 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\") pod \"0d357153-7b81-45fd-9a70-11e58828c720\" (UID: \"0d357153-7b81-45fd-9a70-11e58828c720\") " Nov 02 01:49:13.229920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.229890 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca" pod "0d357153-7b81-45fd-9a70-11e58828c720" (UID: "0d357153-7b81-45fd-9a70-11e58828c720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:13.230248 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.230083 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d357153-7b81-45fd-9a70-11e58828c720-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:13.231615 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.231588 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs" pod "0d357153-7b81-45fd-9a70-11e58828c720" (UID: "0d357153-7b81-45fd-9a70-11e58828c720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:13.231790 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.231770 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2" pod "0d357153-7b81-45fd-9a70-11e58828c720" (UID: "0d357153-7b81-45fd-9a70-11e58828c720"). InnerVolumeSpecName "kube-api-access-w9zg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:13.231839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.231767 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls" pod "0d357153-7b81-45fd-9a70-11e58828c720" (UID: "0d357153-7b81-45fd-9a70-11e58828c720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:13.330876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.330832 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:13.330876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.330875 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9zg2\" (UniqueName: \"kubernetes.io/projected/0d357153-7b81-45fd-9a70-11e58828c720-kube-api-access-w9zg2\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:13.330876 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.330886 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d357153-7b81-45fd-9a70-11e58828c720-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:13.579759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.579725 4552 generic.go:356] "Generic (PLEG): container finished" podID="0d357153-7b81-45fd-9a70-11e58828c720" containerID="af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e" exitCode=0 Nov 02 01:49:13.580268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.579794 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" Nov 02 01:49:13.580268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.579809 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" event={"ID":"0d357153-7b81-45fd-9a70-11e58828c720","Type":"ContainerDied","Data":"af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e"} Nov 02 01:49:13.580268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.579880 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg" event={"ID":"0d357153-7b81-45fd-9a70-11e58828c720","Type":"ContainerDied","Data":"13c7b878eb25a08f2505d24ff9bed96149aa2c5987556574ef93ed83ac52f2ee"} Nov 02 01:49:13.580268 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.579912 4552 scope.go:122] "RemoveContainer" containerID="af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e" Nov 02 01:49:13.591205 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.591183 4552 scope.go:122] "RemoveContainer" containerID="af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e" Nov 02 01:49:13.591461 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:13.591438 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e\": container with ID starting with af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e not found: ID does not exist" containerID="af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e" Nov 02 01:49:13.591527 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.591473 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e"} err="failed to get container status \"af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e\": rpc error: code = NotFound desc = could not find container \"af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e\": container with ID starting with af11666ae8e7190462f33950263494ef4f1dd48bd537116c672113bfe253e79e not found: ID does not exist" Nov 02 01:49:13.606858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.606790 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg"] Nov 02 01:49:13.610351 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:13.610332 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0f32d6-f6cfbf57-trbcg"] Nov 02 01:49:14.044424 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.044392 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d357153-7b81-45fd-9a70-11e58828c720" path="/var/lib/kubelet/pods/0d357153-7b81-45fd-9a70-11e58828c720/volumes" Nov 02 01:49:14.053164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.053137 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:49:14.575596 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.575552 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:14.585162 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.585075 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" containerID="cri-o://8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10" gracePeriod=45 Nov 02 01:49:14.598545 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.598496 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" containerID="cri-o://8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0" gracePeriod=45 Nov 02 01:49:14.696320 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.696279 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 02 01:49:14.880404 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.880378 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:49:14.908321 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.908295 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:49:14.957382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957353 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm\") pod \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " Nov 02 01:49:14.957568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957399 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs\") pod \"997f84c9-3953-470f-bb33-ba1f0375ddd2\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " Nov 02 01:49:14.957568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957475 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x\") pod \"997f84c9-3953-470f-bb33-ba1f0375ddd2\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " Nov 02 01:49:14.957568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957521 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs\") pod \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " Nov 02 01:49:14.957568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957556 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home\") pod \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " Nov 02 01:49:14.957880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957580 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home\") pod \"997f84c9-3953-470f-bb33-ba1f0375ddd2\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " Nov 02 01:49:14.957880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957612 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache\") pod \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " Nov 02 01:49:14.957880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957638 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir\") pod \"997f84c9-3953-470f-bb33-ba1f0375ddd2\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " Nov 02 01:49:14.957880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957776 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm\") pod \"997f84c9-3953-470f-bb33-ba1f0375ddd2\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " Nov 02 01:49:14.957880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957852 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk\" (UniqueName: \"kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk\") pod \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " Nov 02 01:49:14.957880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957878 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir\") pod \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\" (UID: \"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6\") " Nov 02 01:49:14.958294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957876 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home" pod "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" (UID: "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.958294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.957907 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache\") pod \"997f84c9-3953-470f-bb33-ba1f0375ddd2\" (UID: \"997f84c9-3953-470f-bb33-ba1f0375ddd2\") " Nov 02 01:49:14.958294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958001 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache" pod "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" (UID: "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.958294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958223 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home" pod "997f84c9-3953-470f-bb33-ba1f0375ddd2" (UID: "997f84c9-3953-470f-bb33-ba1f0375ddd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.958682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958630 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:14.958682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958651 4552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-home\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:14.958682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958665 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:14.958906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958886 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache" pod "997f84c9-3953-470f-bb33-ba1f0375ddd2" (UID: "997f84c9-3953-470f-bb33-ba1f0375ddd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.958945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958904 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir" pod "997f84c9-3953-470f-bb33-ba1f0375ddd2" (UID: "997f84c9-3953-470f-bb33-ba1f0375ddd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.959003 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.958983 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir" pod "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" (UID: "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.961520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.961487 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm" pod "997f84c9-3953-470f-bb33-ba1f0375ddd2" (UID: "997f84c9-3953-470f-bb33-ba1f0375ddd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.962372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.962346 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk" pod "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" (UID: "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6"). InnerVolumeSpecName "kube-api-access-52swk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:14.963051 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.963034 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs" pod "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" (UID: "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:14.963415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.963262 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm" pod "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" (UID: "0ba69e99-2e61-4fbd-960d-a5dd51e67ab6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:14.964736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.964716 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs" pod "997f84c9-3953-470f-bb33-ba1f0375ddd2" (UID: "997f84c9-3953-470f-bb33-ba1f0375ddd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:14.965269 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:14.965247 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x" pod "997f84c9-3953-470f-bb33-ba1f0375ddd2" (UID: "997f84c9-3953-470f-bb33-ba1f0375ddd2"). InnerVolumeSpecName "kube-api-access-z9s9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:15.059827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059794 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.059827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059820 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52swk\" (UniqueName: \"kubernetes.io/projected/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-kube-api-access-52swk\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.059827 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059830 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.060056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059840 4552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-model-cache\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.060056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059848 4552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-dshm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.060056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059856 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997f84c9-3953-470f-bb33-ba1f0375ddd2-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.060056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059863 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/997f84c9-3953-470f-bb33-ba1f0375ddd2-kube-api-access-z9s9x\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.060056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059871 4552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6-tls-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.060056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.059879 4552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997f84c9-3953-470f-bb33-ba1f0375ddd2-tmp-dir\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:15.600156 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.600110 4552 generic.go:356] "Generic (PLEG): container finished" podID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerID="8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10" exitCode=0 Nov 02 01:49:15.600607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.600185 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" Nov 02 01:49:15.600607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.600216 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" event={"ID":"997f84c9-3953-470f-bb33-ba1f0375ddd2","Type":"ContainerDied","Data":"8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10"} Nov 02 01:49:15.600607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.600267 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l" event={"ID":"997f84c9-3953-470f-bb33-ba1f0375ddd2","Type":"ContainerDied","Data":"c8ef752d252979f731e394e03019cca32bb99acd6e2ce7801a192c17fb6d5f0b"} Nov 02 01:49:15.600607 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.600290 4552 scope.go:122] "RemoveContainer" containerID="8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10" Nov 02 01:49:15.601848 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.601825 4552 generic.go:356] "Generic (PLEG): container finished" podID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerID="8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0" exitCode=0 Nov 02 01:49:15.601964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.601862 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" event={"ID":"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6","Type":"ContainerDied","Data":"8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0"} Nov 02 01:49:15.601964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.601875 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" Nov 02 01:49:15.601964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.601897 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" event={"ID":"0ba69e99-2e61-4fbd-960d-a5dd51e67ab6","Type":"ContainerDied","Data":"77ff51209a60c23563fff857708b7e88d747d38db2dc8ac12588d5c5d493f86e"} Nov 02 01:49:15.619390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.619373 4552 scope.go:122] "RemoveContainer" containerID="8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10" Nov 02 01:49:15.619677 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:15.619657 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10\": container with ID starting with 8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10 not found: ID does not exist" containerID="8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10" Nov 02 01:49:15.619720 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.619685 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10"} err="failed to get container status \"8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10\": rpc error: code = NotFound desc = could not find container \"8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10\": container with ID starting with 8ebcf098aeb4fb94f1f1a68843e8b77425b67da4cce801ae13d554a7c831df10 not found: ID does not exist" Nov 02 01:49:15.619720 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.619704 4552 scope.go:122] "RemoveContainer" containerID="8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0" Nov 02 01:49:15.628389 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.628372 4552 scope.go:122] "RemoveContainer" containerID="8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0" Nov 02 01:49:15.628685 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:15.628662 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0\": container with ID starting with 8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0 not found: ID does not exist" containerID="8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0" Nov 02 01:49:15.628752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.628694 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0"} err="failed to get container status \"8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0\": rpc error: code = NotFound desc = could not find container \"8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0\": container with ID starting with 8019ba0e46857b5dfbe150b7948ab27eb3a092bce6bf6bfd8124efa6c0c464b0 not found: ID does not exist" Nov 02 01:49:15.634770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.634750 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5"] Nov 02 01:49:15.640711 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.640692 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5"] Nov 02 01:49:15.650898 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.650864 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l"] Nov 02 01:49:15.657946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.657925 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-w3-5ed42b-kserve-59d4c5c59b-jqf9l"] Nov 02 01:49:15.804295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:15.804251 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/premium-w3-5ed42b-kserve-5dc6d8d885-484x5" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 02 01:49:16.045456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:16.045420 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" path="/var/lib/kubelet/pods/0ba69e99-2e61-4fbd-960d-a5dd51e67ab6/volumes" Nov 02 01:49:16.046009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:16.045986 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" path="/var/lib/kubelet/pods/997f84c9-3953-470f-bb33-ba1f0375ddd2/volumes" Nov 02 01:49:17.621880 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.621848 4552 generic.go:356] "Generic (PLEG): container finished" podID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerID="76b29ebea7b579d23294e3ccb87fbd29cbc2e5a3af816317efedfa07ee442b53" exitCode=0 Nov 02 01:49:17.622330 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.622027 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" event={"ID":"d1281cd4-30ee-45b9-9c22-932403b8eec2","Type":"ContainerDied","Data":"76b29ebea7b579d23294e3ccb87fbd29cbc2e5a3af816317efedfa07ee442b53"} Nov 02 01:49:17.648984 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.648917 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:49:17.691238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691209 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691274 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691306 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691352 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691406 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691391 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691471 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691523 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691545 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691633 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691578 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert\") pod \"d1281cd4-30ee-45b9-9c22-932403b8eec2\" (UID: \"d1281cd4-30ee-45b9-9c22-932403b8eec2\") " Nov 02 01:49:17.691846 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691674 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.691942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691902 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.691942 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691913 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.692190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.691967 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.692190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.692147 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.692190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.692168 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.692190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.692181 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.692426 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.692194 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.692426 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.692224 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:17.694628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.694284 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "kube-api-access-k7rj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:17.695796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.695745 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:17.696410 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.696386 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.718736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.711828 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo" pod "d1281cd4-30ee-45b9-9c22-932403b8eec2" (UID: "d1281cd4-30ee-45b9-9c22-932403b8eec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:17.792974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.792942 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.792974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.792968 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.792974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.792980 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1281cd4-30ee-45b9-9c22-932403b8eec2-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.793284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.792990 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7rj9\" (UniqueName: \"kubernetes.io/projected/d1281cd4-30ee-45b9-9c22-932403b8eec2-kube-api-access-k7rj9\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.793284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:17.792999 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1281cd4-30ee-45b9-9c22-932403b8eec2-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:18.631111 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:18.631080 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" Nov 02 01:49:18.631604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:18.631110 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl" event={"ID":"d1281cd4-30ee-45b9-9c22-932403b8eec2","Type":"ContainerDied","Data":"d80f64fc7eaf31b7196410f134ba0e1874ee80c30b0b183439a4934f801a3a78"} Nov 02 01:49:18.631604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:18.631215 4552 scope.go:122] "RemoveContainer" containerID="76b29ebea7b579d23294e3ccb87fbd29cbc2e5a3af816317efedfa07ee442b53" Nov 02 01:49:18.663798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:18.663763 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl"] Nov 02 01:49:18.667491 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:18.667460 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-w4qjl"] Nov 02 01:49:20.044188 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:20.044154 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" path="/var/lib/kubelet/pods/d1281cd4-30ee-45b9-9c22-932403b8eec2/volumes" Nov 02 01:49:20.138484 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:20.138447 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:20.858132 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:20.858073 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t"] Nov 02 01:49:20.858391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:20.858360 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" containerID="cri-o://4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060" gracePeriod=30 Nov 02 01:49:21.220019 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.219990 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h"] Nov 02 01:49:21.220654 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.220239 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" podUID="cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" containerName="maas-api" containerID="cri-o://143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6" gracePeriod=30 Nov 02 01:49:21.538145 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.538106 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:49:21.636632 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.636595 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls\") pod \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " Nov 02 01:49:21.636834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.636691 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs\") pod \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " Nov 02 01:49:21.636834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.636799 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca\") pod \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " Nov 02 01:49:21.636834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.636824 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x\" (UniqueName: \"kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x\") pod \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\" (UID: \"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff\") " Nov 02 01:49:21.637514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.637194 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca" pod "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" (UID: "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:21.637514 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.637391 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:21.639111 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.639083 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs" pod "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" (UID: "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:21.639576 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.639549 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls" pod "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" (UID: "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:21.639736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.639716 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x" pod "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" (UID: "cb6322f4-7380-4f9a-bd41-6fa8dfd98bff"). InnerVolumeSpecName "kube-api-access-lm42x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:21.658238 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.658207 4552 generic.go:356] "Generic (PLEG): container finished" podID="cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" containerID="143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6" exitCode=0 Nov 02 01:49:21.658366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.658263 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" Nov 02 01:49:21.658366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.658294 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" event={"ID":"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff","Type":"ContainerDied","Data":"143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6"} Nov 02 01:49:21.658366 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.658362 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h" event={"ID":"cb6322f4-7380-4f9a-bd41-6fa8dfd98bff","Type":"ContainerDied","Data":"3bafa34860a94965ef00c71c7482778211bc8861685b693e97c926641a3c6825"} Nov 02 01:49:21.658540 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.658396 4552 scope.go:122] "RemoveContainer" containerID="143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6" Nov 02 01:49:21.671412 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.671396 4552 scope.go:122] "RemoveContainer" containerID="143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6" Nov 02 01:49:21.671737 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:21.671712 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6\": container with ID starting with 143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6 not found: ID does not exist" containerID="143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6" Nov 02 01:49:21.671804 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.671748 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6"} err="failed to get container status \"143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6\": rpc error: code = NotFound desc = could not find container \"143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6\": container with ID starting with 143c8efb842052dd0a72ea80440fe138c1ef55392ceb20cfe2e6010c8d94c4a6 not found: ID does not exist" Nov 02 01:49:21.687040 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.687007 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h"] Nov 02 01:49:21.689996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.689967 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5ed42b-5c549f5bdd-v2v6h"] Nov 02 01:49:21.739382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.739349 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:21.739382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.739382 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:21.739647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:21.739398 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm42x\" (UniqueName: \"kubernetes.io/projected/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff-kube-api-access-lm42x\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:22.043695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:22.043655 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" path="/var/lib/kubelet/pods/cb6322f4-7380-4f9a-bd41-6fa8dfd98bff/volumes" Nov 02 01:49:23.980218 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.980182 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb"] Nov 02 01:49:23.981057 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981033 4552 cpu_manager.go:414] "containerMap: removing container" podUID="2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" containerName="maas-api" Nov 02 01:49:23.981057 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981058 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981080 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" containerName="main" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981088 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" containerName="main" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981111 4552 cpu_manager.go:414] "containerMap: removing container" podUID="cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981137 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981149 4552 cpu_manager.go:414] "containerMap: removing container" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981156 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981166 4552 cpu_manager.go:414] "containerMap: removing container" podUID="0d357153-7b81-45fd-9a70-11e58828c720" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981173 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d357153-7b81-45fd-9a70-11e58828c720" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981187 4552 cpu_manager.go:414] "containerMap: removing container" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981194 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981213 4552 cpu_manager.go:414] "containerMap: removing container" podUID="fd03964c-70f0-4f6d-b23f-3e4e09650784" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981219 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd03964c-70f0-4f6d-b23f-3e4e09650784" containerName="maas-api" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981241 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" Nov 02 01:49:23.981245 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981248 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981261 4552 cpu_manager.go:414] "containerMap: removing container" podUID="9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981268 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981286 4552 cpu_manager.go:414] "containerMap: removing container" podUID="b7add8ae-4a07-44e6-ae69-be43510e6dce" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981293 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7add8ae-4a07-44e6-ae69-be43510e6dce" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981421 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7add8ae-4a07-44e6-ae69-be43510e6dce" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981440 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7d1de0-c5fa-40b6-9884-6e2cc11217e5" containerName="maas-api" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981450 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b7dc8f-f7e2-4b81-a1e1-884cb20b215e" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981460 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d357153-7b81-45fd-9a70-11e58828c720" containerName="maas-api" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981470 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6322f4-7380-4f9a-bd41-6fa8dfd98bff" containerName="maas-api" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981477 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba69e99-2e61-4fbd-960d-a5dd51e67ab6" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981489 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="997f84c9-3953-470f-bb33-ba1f0375ddd2" containerName="main" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981499 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd03964c-70f0-4f6d-b23f-3e4e09650784" containerName="maas-api" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981510 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1281cd4-30ee-45b9-9c22-932403b8eec2" containerName="istio-proxy" Nov 02 01:49:23.981669 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.981520 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccb7ca1-4b23-4e0f-88bc-9dce000ae5ca" containerName="main" Nov 02 01:49:23.986884 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.986860 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:23.992425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.991486 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:49:23.992425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.991954 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb"] Nov 02 01:49:23.992425 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:23.992083 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:49:24.083604 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.083560 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.083786 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.083720 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcb6\" (UniqueName: \"kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.083875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.083855 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.184528 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.184489 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.184679 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.184595 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.184757 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.184694 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcb6\" (UniqueName: \"kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.185468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.185443 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.187082 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.187057 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.198272 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.198248 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcb6\" (UniqueName: \"kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6\") pod \"maas-api-revoke-sub-ef767fd1-6xssb\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.307482 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.307409 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:24.573207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.573095 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:24.612705 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.612681 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb"] Nov 02 01:49:24.613561 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:49:24.613537 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27359d1d_4991_417c_b1c3_33a016b58101.slice/crio-490c310e9ad128243f47cbe7eb6696b733bf8e3fb53f42bf4a36607fbb027d83 WatchSource:0}: Error finding container 490c310e9ad128243f47cbe7eb6696b733bf8e3fb53f42bf4a36607fbb027d83: Status 404 returned error can't find the container with id 490c310e9ad128243f47cbe7eb6696b733bf8e3fb53f42bf4a36607fbb027d83 Nov 02 01:49:24.615572 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.615550 4552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:49:24.688440 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:24.688402 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" event={"ID":"27359d1d-4991-417c-b1c3-33a016b58101","Type":"ContainerStarted","Data":"490c310e9ad128243f47cbe7eb6696b733bf8e3fb53f42bf4a36607fbb027d83"} Nov 02 01:49:25.561992 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.561931 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:49:25.696344 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.696304 4552 generic.go:356] "Generic (PLEG): container finished" podID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerID="c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc" exitCode=0 Nov 02 01:49:25.696490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.696353 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" Nov 02 01:49:25.696490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.696365 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" event={"ID":"4d34a683-dd9e-4bb4-8b63-f3e5249e523e","Type":"ContainerDied","Data":"c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc"} Nov 02 01:49:25.696490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.696427 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k" event={"ID":"4d34a683-dd9e-4bb4-8b63-f3e5249e523e","Type":"ContainerDied","Data":"581eec7eefa8782706bbb480cc15e2b9c10c0506eac885f4ba0cd4db586c1eae"} Nov 02 01:49:25.696490 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.696457 4552 scope.go:122] "RemoveContainer" containerID="c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc" Nov 02 01:49:25.701334 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.701311 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume\") pod \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " Nov 02 01:49:25.701457 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.701381 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9\" (UniqueName: \"kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9\") pod \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\" (UID: \"4d34a683-dd9e-4bb4-8b63-f3e5249e523e\") " Nov 02 01:49:25.701686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.701666 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume" pod "4d34a683-dd9e-4bb4-8b63-f3e5249e523e" (UID: "4d34a683-dd9e-4bb4-8b63-f3e5249e523e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:25.701795 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.701773 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:25.703769 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.703743 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9" pod "4d34a683-dd9e-4bb4-8b63-f3e5249e523e" (UID: "4d34a683-dd9e-4bb4-8b63-f3e5249e523e"). InnerVolumeSpecName "kube-api-access-5ffd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:25.706584 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.706562 4552 scope.go:122] "RemoveContainer" containerID="c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc" Nov 02 01:49:25.706889 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:25.706868 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc\": container with ID starting with c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc not found: ID does not exist" containerID="c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc" Nov 02 01:49:25.706947 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.706896 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc"} err="failed to get container status \"c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc\": rpc error: code = NotFound desc = could not find container \"c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc\": container with ID starting with c8eabc3f3e1254039c8e082cd6c062d2670864ba3568a8b99269bf2dae548cfc not found: ID does not exist" Nov 02 01:49:25.803178 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:25.803141 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ffd9\" (UniqueName: \"kubernetes.io/projected/4d34a683-dd9e-4bb4-8b63-f3e5249e523e-kube-api-access-5ffd9\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.073227 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.073190 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k"] Nov 02 01:49:26.076215 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.076185 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d4a0-657d89cbf9-p559k"] Nov 02 01:49:26.185386 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.184645 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:49:26.327630 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327594 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327643 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327692 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327793 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327725 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327831 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327878 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327914 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.327967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327933 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.327967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.327947 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.328288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.328006 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.328288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.328071 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert\") pod \"40a50d09-f5e3-4ddb-9955-248318b2082f\" (UID: \"40a50d09-f5e3-4ddb-9955-248318b2082f\") " Nov 02 01:49:26.328288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.328171 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.328448 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.328350 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.328507 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.328441 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:26.329937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.329908 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.330069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.329942 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.330069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.329957 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a50d09-f5e3-4ddb-9955-248318b2082f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.330069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.329981 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.330069 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.329997 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.330640 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.330616 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.330722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.330648 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:26.330959 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.330931 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:26.352882 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.352836 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9" pod "40a50d09-f5e3-4ddb-9955-248318b2082f" (UID: "40a50d09-f5e3-4ddb-9955-248318b2082f"). InnerVolumeSpecName "kube-api-access-6k8b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:26.431511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.431412 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.431511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.431462 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.431511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.431472 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a50d09-f5e3-4ddb-9955-248318b2082f-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.431511 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.431483 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8b9\" (UniqueName: \"kubernetes.io/projected/40a50d09-f5e3-4ddb-9955-248318b2082f-kube-api-access-6k8b9\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.707159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.707052 4552 generic.go:356] "Generic (PLEG): container finished" podID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerID="4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060" exitCode=0 Nov 02 01:49:26.707159 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.707156 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" Nov 02 01:49:26.707686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.707151 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" event={"ID":"40a50d09-f5e3-4ddb-9955-248318b2082f","Type":"ContainerDied","Data":"4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060"} Nov 02 01:49:26.707686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.707238 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t" event={"ID":"40a50d09-f5e3-4ddb-9955-248318b2082f","Type":"ContainerDied","Data":"6b0b318c98724e309f9fec821ec8235c439ab2ec22853766f70e69ba8676903e"} Nov 02 01:49:26.707686 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.707270 4552 scope.go:122] "RemoveContainer" containerID="4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060" Nov 02 01:49:26.720736 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.720717 4552 scope.go:122] "RemoveContainer" containerID="4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060" Nov 02 01:49:26.721082 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:26.721052 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060\": container with ID starting with 4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060 not found: ID does not exist" containerID="4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060" Nov 02 01:49:26.721233 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.721096 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060"} err="failed to get container status \"4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060\": rpc error: code = NotFound desc = could not find container \"4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060\": container with ID starting with 4bf6aa9bff52d8c49951fa6d9f9599b0c910c10e6585c46aa433fd39d2f98060 not found: ID does not exist" Nov 02 01:49:26.737372 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.737347 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t"] Nov 02 01:49:26.741999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:26.741973 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-7cd9t"] Nov 02 01:49:27.753532 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:27.753492 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:49:28.046907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:28.046731 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" path="/var/lib/kubelet/pods/40a50d09-f5e3-4ddb-9955-248318b2082f/volumes" Nov 02 01:49:28.047331 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:28.047309 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" path="/var/lib/kubelet/pods/4d34a683-dd9e-4bb4-8b63-f3e5249e523e/volumes" Nov 02 01:49:28.725647 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:28.725612 4552 generic.go:356] "Generic (PLEG): container finished" podID="27359d1d-4991-417c-b1c3-33a016b58101" containerID="3f8bf205f26c3695c6ecce523dacc1272e15938d316df0142c3f21be2b6ea7ee" exitCode=0 Nov 02 01:49:28.725824 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:28.725698 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" event={"ID":"27359d1d-4991-417c-b1c3-33a016b58101","Type":"ContainerDied","Data":"3f8bf205f26c3695c6ecce523dacc1272e15938d316df0142c3f21be2b6ea7ee"} Nov 02 01:49:29.009834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.009795 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c"] Nov 02 01:49:29.010470 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.010019 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" containerID="cri-o://c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd" gracePeriod=30 Nov 02 01:49:29.896391 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.896362 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:29.992974 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.992879 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6\" (UniqueName: \"kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6\") pod \"27359d1d-4991-417c-b1c3-33a016b58101\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " Nov 02 01:49:29.993203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.992987 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca\") pod \"27359d1d-4991-417c-b1c3-33a016b58101\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " Nov 02 01:49:29.993203 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.993048 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca\") pod \"27359d1d-4991-417c-b1c3-33a016b58101\" (UID: \"27359d1d-4991-417c-b1c3-33a016b58101\") " Nov 02 01:49:29.993392 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.993366 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca" pod "27359d1d-4991-417c-b1c3-33a016b58101" (UID: "27359d1d-4991-417c-b1c3-33a016b58101"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:29.993672 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.993638 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:29.995408 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.995386 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca" pod "27359d1d-4991-417c-b1c3-33a016b58101" (UID: "27359d1d-4991-417c-b1c3-33a016b58101"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:29.995602 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:29.995578 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6" pod "27359d1d-4991-417c-b1c3-33a016b58101" (UID: "27359d1d-4991-417c-b1c3-33a016b58101"). InnerVolumeSpecName "kube-api-access-xkcb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:30.094799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.094764 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27359d1d-4991-417c-b1c3-33a016b58101-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:30.094799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.094797 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkcb6\" (UniqueName: \"kubernetes.io/projected/27359d1d-4991-417c-b1c3-33a016b58101-kube-api-access-xkcb6\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:30.137833 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.137794 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:30.397896 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.397868 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:49:30.498834 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.498800 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume\") pod \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " Nov 02 01:49:30.499021 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.498980 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5\" (UniqueName: \"kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5\") pod \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\" (UID: \"4efc4e7e-a049-4397-9fcf-2eae203cb81b\") " Nov 02 01:49:30.499276 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.499252 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume" pod "4efc4e7e-a049-4397-9fcf-2eae203cb81b" (UID: "4efc4e7e-a049-4397-9fcf-2eae203cb81b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:30.500023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.499997 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efc4e7e-a049-4397-9fcf-2eae203cb81b-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:30.509785 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.509643 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5" pod "4efc4e7e-a049-4397-9fcf-2eae203cb81b" (UID: "4efc4e7e-a049-4397-9fcf-2eae203cb81b"). InnerVolumeSpecName "kube-api-access-8n2w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:30.601477 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.601441 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2w5\" (UniqueName: \"kubernetes.io/projected/4efc4e7e-a049-4397-9fcf-2eae203cb81b-kube-api-access-8n2w5\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:30.742817 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.742771 4552 generic.go:356] "Generic (PLEG): container finished" podID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerID="bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887" exitCode=0 Nov 02 01:49:30.743013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.742833 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" Nov 02 01:49:30.743013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.742858 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" event={"ID":"4efc4e7e-a049-4397-9fcf-2eae203cb81b","Type":"ContainerDied","Data":"bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887"} Nov 02 01:49:30.743013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.742966 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc" event={"ID":"4efc4e7e-a049-4397-9fcf-2eae203cb81b","Type":"ContainerDied","Data":"62d56f0f6bc37b2057f0977bba75fe1f48fa3494f379acbe00aa67c14d7b6625"} Nov 02 01:49:30.743013 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.742999 4552 scope.go:122] "RemoveContainer" containerID="bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887" Nov 02 01:49:30.744689 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.744670 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" event={"ID":"27359d1d-4991-417c-b1c3-33a016b58101","Type":"ContainerDied","Data":"490c310e9ad128243f47cbe7eb6696b733bf8e3fb53f42bf4a36607fbb027d83"} Nov 02 01:49:30.744842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.744694 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb" Nov 02 01:49:30.744842 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.744712 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490c310e9ad128243f47cbe7eb6696b733bf8e3fb53f42bf4a36607fbb027d83" Nov 02 01:49:30.757515 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.757492 4552 scope.go:122] "RemoveContainer" containerID="bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887" Nov 02 01:49:30.757913 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:30.757890 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887\": container with ID starting with bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887 not found: ID does not exist" containerID="bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887" Nov 02 01:49:30.757978 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.757926 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887"} err="failed to get container status \"bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887\": rpc error: code = NotFound desc = could not find container \"bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887\": container with ID starting with bef9448ae113f9850e42bb853c909182be4d1dcf22e1478920b5d63a84912887 not found: ID does not exist" Nov 02 01:49:30.775932 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.775855 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc"] Nov 02 01:49:30.781318 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:30.781292 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5ed42b-8c76d6967-mh7nc"] Nov 02 01:49:31.840045 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.840012 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj"] Nov 02 01:49:31.919416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.919381 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv"] Nov 02 01:49:31.920306 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920284 4552 cpu_manager.go:414] "containerMap: removing container" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerName="payload-pre-processing" Nov 02 01:49:31.920387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920310 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerName="payload-pre-processing" Nov 02 01:49:31.920387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920327 4552 cpu_manager.go:414] "containerMap: removing container" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" Nov 02 01:49:31.920387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920336 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" Nov 02 01:49:31.920387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920352 4552 cpu_manager.go:414] "containerMap: removing container" podUID="27359d1d-4991-417c-b1c3-33a016b58101" containerName="revoke-keys" Nov 02 01:49:31.920387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920360 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27359d1d-4991-417c-b1c3-33a016b58101" containerName="revoke-keys" Nov 02 01:49:31.920646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920390 4552 cpu_manager.go:414] "containerMap: removing container" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerName="payload-processing" Nov 02 01:49:31.920646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920398 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerName="payload-processing" Nov 02 01:49:31.920646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920556 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" containerName="payload-pre-processing" Nov 02 01:49:31.920646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920570 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a50d09-f5e3-4ddb-9955-248318b2082f" containerName="istio-proxy" Nov 02 01:49:31.920646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920580 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="27359d1d-4991-417c-b1c3-33a016b58101" containerName="revoke-keys" Nov 02 01:49:31.920646 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.920593 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d34a683-dd9e-4bb4-8b63-f3e5249e523e" containerName="payload-processing" Nov 02 01:49:31.926628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.926601 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:31.939090 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.939060 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:49:31.939304 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.939237 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:49:31.942438 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:31.942409 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv"] Nov 02 01:49:32.044333 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.044295 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efc4e7e-a049-4397-9fcf-2eae203cb81b" path="/var/lib/kubelet/pods/4efc4e7e-a049-4397-9fcf-2eae203cb81b/volumes" Nov 02 01:49:32.118729 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.118649 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.118896 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.118881 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.118938 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.118925 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.220468 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.220420 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.220676 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.220508 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.220676 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.220547 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.221174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.221111 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.223251 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.223228 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.234285 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.234255 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw\") pod \"maas-api-revoke-sub-6e073df1-896sv\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.249760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.249735 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:32.531471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.531443 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv"] Nov 02 01:49:32.532558 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:49:32.532537 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558b782a_4d28_4e2e_8664_1c1f4a02e4dc.slice/crio-7ed57ab6849b1d049113bb7c24e629166feab2f272c3fb86dbfff0254184c88f WatchSource:0}: Error finding container 7ed57ab6849b1d049113bb7c24e629166feab2f272c3fb86dbfff0254184c88f: Status 404 returned error can't find the container with id 7ed57ab6849b1d049113bb7c24e629166feab2f272c3fb86dbfff0254184c88f Nov 02 01:49:32.760948 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:32.760910 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" event={"ID":"558b782a-4d28-4e2e-8664-1c1f4a02e4dc","Type":"ContainerStarted","Data":"7ed57ab6849b1d049113bb7c24e629166feab2f272c3fb86dbfff0254184c88f"} Nov 02 01:49:33.054692 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:33.054648 4552 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:49:33.055195 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:33.054720 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:49:33.770220 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:33.770182 4552 generic.go:356] "Generic (PLEG): container finished" podID="558b782a-4d28-4e2e-8664-1c1f4a02e4dc" containerID="c276c407b7128fe6d959ec0bf7bece907c04bea6569cf446c5886c6087ccb6bd" exitCode=0 Nov 02 01:49:33.770407 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:33.770274 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" event={"ID":"558b782a-4d28-4e2e-8664-1c1f4a02e4dc","Type":"ContainerDied","Data":"c276c407b7128fe6d959ec0bf7bece907c04bea6569cf446c5886c6087ccb6bd"} Nov 02 01:49:34.294806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.294743 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:49:34.342755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.342719 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.342961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.342818 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.342961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.342877 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.342961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.342911 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.342961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.342949 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.343216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343076 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.343216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343110 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.343216 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343201 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.343369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343247 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.343369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343285 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.343369 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343352 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm\") pod \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\" (UID: \"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11\") " Nov 02 01:49:34.343519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343470 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.343519 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343491 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:34.343838 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343815 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.344009 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.343993 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.344149 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.344108 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.344271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.344257 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.344595 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.344573 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.348548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.348488 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:34.349526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.349485 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "kube-api-access-n2bkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:34.350722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.350006 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.372750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.372705 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo" pod "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" (UID: "f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:34.445513 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.445473 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.445513 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.445516 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2bkm\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-kube-api-access-n2bkm\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.445740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.445532 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.445740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.445546 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.445740 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.445559 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.782395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.782357 4552 generic.go:356] "Generic (PLEG): container finished" podID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerID="c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd" exitCode=0 Nov 02 01:49:34.782395 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.782401 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" Nov 02 01:49:34.782682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.782438 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" event={"ID":"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11","Type":"ContainerDied","Data":"c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd"} Nov 02 01:49:34.782682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.782503 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c" event={"ID":"f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11","Type":"ContainerDied","Data":"d46406bf436d069901a48e63b210a73653d13ada1bf2c41d5b97c2975e17b937"} Nov 02 01:49:34.782682 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.782535 4552 scope.go:122] "RemoveContainer" containerID="c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd" Nov 02 01:49:34.797471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.797450 4552 scope.go:122] "RemoveContainer" containerID="c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd" Nov 02 01:49:34.797886 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:34.797852 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd\": container with ID starting with c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd not found: ID does not exist" containerID="c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd" Nov 02 01:49:34.797996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.797898 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd"} err="failed to get container status \"c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd\": rpc error: code = NotFound desc = could not find container \"c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd\": container with ID starting with c58345b0ec9d0b5948dfdf447c9bca0e563032a06a0eb8790e94f6f1568702cd not found: ID does not exist" Nov 02 01:49:34.813504 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.813457 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c"] Nov 02 01:49:34.816860 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.816829 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9tz6c"] Nov 02 01:49:34.963022 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:34.962994 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:35.052168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.052067 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw\") pod \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " Nov 02 01:49:35.052168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.052150 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca\") pod \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " Nov 02 01:49:35.052423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.052243 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca\") pod \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\" (UID: \"558b782a-4d28-4e2e-8664-1c1f4a02e4dc\") " Nov 02 01:49:35.052675 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.052645 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca" pod "558b782a-4d28-4e2e-8664-1c1f4a02e4dc" (UID: "558b782a-4d28-4e2e-8664-1c1f4a02e4dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:35.054562 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.054532 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw" pod "558b782a-4d28-4e2e-8664-1c1f4a02e4dc" (UID: "558b782a-4d28-4e2e-8664-1c1f4a02e4dc"). InnerVolumeSpecName "kube-api-access-6tgvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:35.054562 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.054545 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca" pod "558b782a-4d28-4e2e-8664-1c1f4a02e4dc" (UID: "558b782a-4d28-4e2e-8664-1c1f4a02e4dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:35.153875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.153833 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:35.153875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.153873 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:35.154163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.153889 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/558b782a-4d28-4e2e-8664-1c1f4a02e4dc-kube-api-access-6tgvw\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:35.517270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.517235 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:49:35.791267 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.791186 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" Nov 02 01:49:35.791267 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.791185 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv" event={"ID":"558b782a-4d28-4e2e-8664-1c1f4a02e4dc","Type":"ContainerDied","Data":"7ed57ab6849b1d049113bb7c24e629166feab2f272c3fb86dbfff0254184c88f"} Nov 02 01:49:35.791267 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:35.791243 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ed57ab6849b1d049113bb7c24e629166feab2f272c3fb86dbfff0254184c88f" Nov 02 01:49:36.043176 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:36.043082 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" path="/var/lib/kubelet/pods/f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11/volumes" Nov 02 01:49:36.840508 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:36.840462 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerName="payload-processing" containerID="cri-o://182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba" gracePeriod=25 Nov 02 01:49:38.122621 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:38.122578 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:41.368051 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:41.368012 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2"] Nov 02 01:49:41.369379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:41.368909 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" containerID="cri-o://aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f" gracePeriod=30 Nov 02 01:49:46.646261 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.645880 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:49:46.692758 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.692719 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.692939 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.692790 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.692939 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.692867 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.692939 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.692907 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.692939 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.692932 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.693172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.692973 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.693172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.693010 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.693172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.693065 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.693310 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.693215 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token\") pod \"37a89abf-f2a4-4fcc-8129-337130c09ad5\" (UID: \"37a89abf-f2a4-4fcc-8129-337130c09ad5\") " Nov 02 01:49:46.693441 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.693401 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.693719 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.693670 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.693943 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.693915 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.694179 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.694140 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.694858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.694426 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:46.695416 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.695196 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.698026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.697980 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.717206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.704952 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "kube-api-access-lcc6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:46.717206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.705246 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:46.717206 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.707056 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo" pod "37a89abf-f2a4-4fcc-8129-337130c09ad5" (UID: "37a89abf-f2a4-4fcc-8129-337130c09ad5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794770 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794807 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794823 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794836 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a89abf-f2a4-4fcc-8129-337130c09ad5-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794850 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794864 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcc6p\" (UniqueName: \"kubernetes.io/projected/37a89abf-f2a4-4fcc-8129-337130c09ad5-kube-api-access-lcc6p\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794879 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.794920 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.794891 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a89abf-f2a4-4fcc-8129-337130c09ad5-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.874288 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.874247 4552 generic.go:356] "Generic (PLEG): container finished" podID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerID="aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f" exitCode=0 Nov 02 01:49:46.874471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.874301 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" Nov 02 01:49:46.874471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.874330 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" event={"ID":"37a89abf-f2a4-4fcc-8129-337130c09ad5","Type":"ContainerDied","Data":"aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f"} Nov 02 01:49:46.874471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.874392 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2" event={"ID":"37a89abf-f2a4-4fcc-8129-337130c09ad5","Type":"ContainerDied","Data":"cada15b5bd8f006da0f8a5ca367a9ed43629647dc1b7e3d0960cd32b9fc83a55"} Nov 02 01:49:46.874471 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.874423 4552 scope.go:122] "RemoveContainer" containerID="aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f" Nov 02 01:49:46.885295 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.885270 4552 scope.go:122] "RemoveContainer" containerID="aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f" Nov 02 01:49:46.885645 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:46.885619 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f\": container with ID starting with aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f not found: ID does not exist" containerID="aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f" Nov 02 01:49:46.885751 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.885655 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f"} err="failed to get container status \"aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f\": rpc error: code = NotFound desc = could not find container \"aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f\": container with ID starting with aad39264d4fa8ebb4a2d015286173449ac9fc22fa8434c1b8325aed30f36ff1f not found: ID does not exist" Nov 02 01:49:46.902168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.902064 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2"] Nov 02 01:49:46.906174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:46.906105 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-fjrd2"] Nov 02 01:49:48.045741 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:48.045710 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" path="/var/lib/kubelet/pods/37a89abf-f2a4-4fcc-8129-337130c09ad5/volumes" Nov 02 01:49:48.122837 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:48.122794 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:51.646373 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:51.646337 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:49:53.871164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:53.868713 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7"] Nov 02 01:49:53.871164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:53.869331 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" containerID="cri-o://8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771" gracePeriod=30 Nov 02 01:49:57.097520 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.097461 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:49:57.230798 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.230764 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume\") pod \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " Nov 02 01:49:57.231011 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.230889 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd\" (UniqueName: \"kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd\") pod \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\" (UID: \"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c\") " Nov 02 01:49:57.231270 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.231240 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume" pod "1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" (UID: "1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:57.233415 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.233390 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd" pod "1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" (UID: "1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c"). InnerVolumeSpecName "kube-api-access-4d4fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:57.332839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.332806 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4fd\" (UniqueName: \"kubernetes.io/projected/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-kube-api-access-4d4fd\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:57.332839 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.332835 4552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c-plugins-config-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:57.721134 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.721077 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:49:57.753885 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:57.753846 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:49:57.952830 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.952792 4552 generic.go:356] "Generic (PLEG): container finished" podID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerID="182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba" exitCode=0 Nov 02 01:49:57.953034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.952840 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" Nov 02 01:49:57.953034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.952876 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" event={"ID":"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c","Type":"ContainerDied","Data":"182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba"} Nov 02 01:49:57.953034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.952931 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj" event={"ID":"1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c","Type":"ContainerDied","Data":"19c7c36c851280899e0a9df5c2f1e14fb0084999c7739fdc614da509eb7606db"} Nov 02 01:49:57.953034 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.952954 4552 scope.go:122] "RemoveContainer" containerID="182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba" Nov 02 01:49:57.963983 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.963948 4552 scope.go:122] "RemoveContainer" containerID="182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba" Nov 02 01:49:57.964361 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:49:57.964338 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba\": container with ID starting with 182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba not found: ID does not exist" containerID="182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba" Nov 02 01:49:57.964443 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.964371 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba"} err="failed to get container status \"182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba\": rpc error: code = NotFound desc = could not find container \"182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba\": container with ID starting with 182568fb0436074311c3e7ac64a66599fb89d0ef7ea06040150a4b522ff6c9ba not found: ID does not exist" Nov 02 01:49:57.977999 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.977925 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj"] Nov 02 01:49:57.984442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:57.984416 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-123986-594d7c5d55-7sjqj"] Nov 02 01:49:58.003186 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003148 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587977945b-vnblh"] Nov 02 01:49:58.003796 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003776 4552 cpu_manager.go:414] "containerMap: removing container" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003798 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003854 4552 cpu_manager.go:414] "containerMap: removing container" podUID="558b782a-4d28-4e2e-8664-1c1f4a02e4dc" containerName="revoke-keys" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003863 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558b782a-4d28-4e2e-8664-1c1f4a02e4dc" containerName="revoke-keys" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003878 4552 cpu_manager.go:414] "containerMap: removing container" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003885 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003897 4552 cpu_manager.go:414] "containerMap: removing container" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerName="payload-processing" Nov 02 01:49:58.003937 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.003905 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerName="payload-processing" Nov 02 01:49:58.004349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.004008 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="558b782a-4d28-4e2e-8664-1c1f4a02e4dc" containerName="revoke-keys" Nov 02 01:49:58.004349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.004025 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a89abf-f2a4-4fcc-8129-337130c09ad5" containerName="istio-proxy" Nov 02 01:49:58.004349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.004034 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" containerName="payload-processing" Nov 02 01:49:58.004349 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.004044 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a3e4e7-a07d-4e1b-ba4b-ef74f0a3fe11" containerName="istio-proxy" Nov 02 01:49:58.008770 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.008748 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.016916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.016421 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 01:49:58.016916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.016483 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 01:49:58.016916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.016545 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 01:49:58.016916 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.016601 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldcrm\"" Nov 02 01:49:58.018073 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.018051 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587977945b-vnblh"] Nov 02 01:49:58.045194 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.045161 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c" path="/var/lib/kubelet/pods/1a8506fe-c38f-4ed3-aaad-e7f86fc64c6c/volumes" Nov 02 01:49:58.142510 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.142463 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.143067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.142544 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fccj\" (UniqueName: \"kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.143067 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.142708 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.244184 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.244063 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.244542 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.244516 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.244723 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.244704 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fccj\" (UniqueName: \"kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.254357 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.253155 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.261024 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.260971 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.261828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.261686 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fccj\" (UniqueName: \"kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj\") pod \"maas-controller-587977945b-vnblh\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.324644 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.324611 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:49:58.622190 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.622157 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587977945b-vnblh"] Nov 02 01:49:58.623924 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:49:58.623892 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2742ecf_252d_419f_98e6_35b3f9e37e96.slice/crio-20b50c6b784a6653ecb342c83d285de307b26830a7acc3be2e2ebea30d479c65 WatchSource:0}: Error finding container 20b50c6b784a6653ecb342c83d285de307b26830a7acc3be2e2ebea30d479c65: Status 404 returned error can't find the container with id 20b50c6b784a6653ecb342c83d285de307b26830a7acc3be2e2ebea30d479c65 Nov 02 01:49:58.965611 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:58.965565 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-vnblh" event={"ID":"a2742ecf-252d-419f-98e6-35b3f9e37e96","Type":"ContainerStarted","Data":"20b50c6b784a6653ecb342c83d285de307b26830a7acc3be2e2ebea30d479c65"} Nov 02 01:49:59.562148 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.562040 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:49:59.681493 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.681452 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.681872 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.681849 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.682026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.682009 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.682301 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.682278 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.682456 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.682439 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.682565 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.682551 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.686208 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.686183 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.686362 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.686347 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.686568 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.686554 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket\") pod \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\" (UID: \"d8e0813c-0557-4448-9cd2-ee5c591cae4a\") " Nov 02 01:49:59.700164 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.696493 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.725486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.725457 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.726280 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.726201 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.726722 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.726637 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:59.726918 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.726861 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.735235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.730568 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:59.735526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.735488 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "kube-api-access-bjc25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:59.789806 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.789765 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.809954 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.809929 4552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-token\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810168 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810148 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjc25\" (UniqueName: \"kubernetes.io/projected/d8e0813c-0557-4448-9cd2-ee5c591cae4a-kube-api-access-bjc25\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810300 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810285 4552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-envoy\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810419 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810405 4552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-data\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810532 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810520 4552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istiod-ca-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810619 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810608 4552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810716 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810704 4552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-credential-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.810828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.810814 4552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e0813c-0557-4448-9cd2-ee5c591cae4a-workload-socket\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.813002 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.812943 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo" pod "d8e0813c-0557-4448-9cd2-ee5c591cae4a" (UID: "d8e0813c-0557-4448-9cd2-ee5c591cae4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:59.916170 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:49:59.916139 4552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e0813c-0557-4448-9cd2-ee5c591cae4a-istio-podinfo\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:00.023143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.021951 4552 generic.go:356] "Generic (PLEG): container finished" podID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerID="8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771" exitCode=0 Nov 02 01:50:00.023143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.022283 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" event={"ID":"d8e0813c-0557-4448-9cd2-ee5c591cae4a","Type":"ContainerDied","Data":"8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771"} Nov 02 01:50:00.023143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.022383 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" event={"ID":"d8e0813c-0557-4448-9cd2-ee5c591cae4a","Type":"ContainerDied","Data":"19b3f1cbf0ccc4d43395d5522cbe1a6d518110a1aa5eb970339e0fc2e4cb2845"} Nov 02 01:50:00.023143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.022418 4552 scope.go:122] "RemoveContainer" containerID="8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771" Nov 02 01:50:00.023143 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.022567 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7" Nov 02 01:50:00.099235 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.087558 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-vnblh" event={"ID":"a2742ecf-252d-419f-98e6-35b3f9e37e96","Type":"ContainerStarted","Data":"a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594"} Nov 02 01:50:00.099551 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.099522 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:50:00.124750 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.124687 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-587977945b-vnblh" podStartSLOduration=2.050764464 podStartE2EDuration="3.124666281s" totalImagesPullingTime="1.073901817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:00.106659643 +0000 UTC m=+2100.212448559" watchObservedRunningTime="2026-11-02 01:50:00.124666281 +0000 UTC m=+2100.230455196" Nov 02 01:50:00.149718 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.146970 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7"] Nov 02 01:50:00.200382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.174425 4552 scope.go:122] "RemoveContainer" containerID="8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771" Nov 02 01:50:00.200382 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:50:00.174865 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771\": container with ID starting with 8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771 not found: ID does not exist" containerID="8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771" Nov 02 01:50:00.200382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.174903 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771"} err="failed to get container status \"8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771\": rpc error: code = NotFound desc = could not find container \"8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771\": container with ID starting with 8896b9f01c8472b9a1d3271aa028615d4c0a5311a7cbd90e9ba4f16f9a4fa771 not found: ID does not exist" Nov 02 01:50:00.200382 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:00.197493 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-2qpx7"] Nov 02 01:50:02.043152 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:02.043076 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" path="/var/lib/kubelet/pods/d8e0813c-0557-4448-9cd2-ee5c591cae4a/volumes" Nov 02 01:50:11.066290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:11.066205 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:50:18.953103 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:18.953063 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:50:27.751736 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:50:27.751693 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:50:29.287454 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.287418 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587977945b-vnblh"] Nov 02 01:50:29.288439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.288055 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-587977945b-vnblh" podUID="a2742ecf-252d-419f-98e6-35b3f9e37e96" containerName="manager" containerID="cri-o://a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594" gracePeriod=30 Nov 02 01:50:29.570945 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.570916 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:50:29.672557 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.672522 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj\" (UniqueName: \"kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj\") pod \"a2742ecf-252d-419f-98e6-35b3f9e37e96\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " Nov 02 01:50:29.672760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.672613 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert\") pod \"a2742ecf-252d-419f-98e6-35b3f9e37e96\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " Nov 02 01:50:29.672760 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.672680 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs\") pod \"a2742ecf-252d-419f-98e6-35b3f9e37e96\" (UID: \"a2742ecf-252d-419f-98e6-35b3f9e37e96\") " Nov 02 01:50:29.674973 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.674941 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj" pod "a2742ecf-252d-419f-98e6-35b3f9e37e96" (UID: "a2742ecf-252d-419f-98e6-35b3f9e37e96"). InnerVolumeSpecName "kube-api-access-2fccj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:50:29.675108 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.675002 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert" pod "a2742ecf-252d-419f-98e6-35b3f9e37e96" (UID: "a2742ecf-252d-419f-98e6-35b3f9e37e96"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:50:29.675174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.675140 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs" pod "a2742ecf-252d-419f-98e6-35b3f9e37e96" (UID: "a2742ecf-252d-419f-98e6-35b3f9e37e96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:50:29.773964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.773919 4552 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-webhook-cert\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:29.773964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.773957 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2742ecf-252d-419f-98e6-35b3f9e37e96-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:29.773964 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:29.773967 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fccj\" (UniqueName: \"kubernetes.io/projected/a2742ecf-252d-419f-98e6-35b3f9e37e96-kube-api-access-2fccj\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:30.262066 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.262032 4552 generic.go:356] "Generic (PLEG): container finished" podID="a2742ecf-252d-419f-98e6-35b3f9e37e96" containerID="a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594" exitCode=0 Nov 02 01:50:30.262258 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.262079 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-vnblh" Nov 02 01:50:30.262258 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.262107 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-vnblh" event={"ID":"a2742ecf-252d-419f-98e6-35b3f9e37e96","Type":"ContainerDied","Data":"a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594"} Nov 02 01:50:30.262258 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.262176 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-vnblh" event={"ID":"a2742ecf-252d-419f-98e6-35b3f9e37e96","Type":"ContainerDied","Data":"20b50c6b784a6653ecb342c83d285de307b26830a7acc3be2e2ebea30d479c65"} Nov 02 01:50:30.262258 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.262197 4552 scope.go:122] "RemoveContainer" containerID="a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594" Nov 02 01:50:30.271628 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.271603 4552 scope.go:122] "RemoveContainer" containerID="a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594" Nov 02 01:50:30.271933 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:50:30.271878 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594\": container with ID starting with a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594 not found: ID does not exist" containerID="a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594" Nov 02 01:50:30.272012 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.271945 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594"} err="failed to get container status \"a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594\": rpc error: code = NotFound desc = could not find container \"a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594\": container with ID starting with a00540f7d5585d8e9400aa15cf0f10cde09277ed7bc87b5e522de7efaa5f6594 not found: ID does not exist" Nov 02 01:50:30.282294 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.282262 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587977945b-vnblh"] Nov 02 01:50:30.286363 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:30.286339 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-587977945b-vnblh"] Nov 02 01:50:31.581907 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.581870 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587977945b-p7s6j"] Nov 02 01:50:31.582439 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.582423 4552 cpu_manager.go:414] "containerMap: removing container" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" Nov 02 01:50:31.582489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.582442 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" Nov 02 01:50:31.582489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.582473 4552 cpu_manager.go:414] "containerMap: removing container" podUID="a2742ecf-252d-419f-98e6-35b3f9e37e96" containerName="manager" Nov 02 01:50:31.582489 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.582478 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2742ecf-252d-419f-98e6-35b3f9e37e96" containerName="manager" Nov 02 01:50:31.582582 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.582547 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e0813c-0557-4448-9cd2-ee5c591cae4a" containerName="istio-proxy" Nov 02 01:50:31.582582 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.582563 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2742ecf-252d-419f-98e6-35b3f9e37e96" containerName="manager" Nov 02 01:50:31.587505 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.587479 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.592752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.592718 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 01:50:31.597264 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.597235 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldcrm\"" Nov 02 01:50:31.597423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.597280 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 01:50:31.597423 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.597361 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 01:50:31.600746 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.600725 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587977945b-p7s6j"] Nov 02 01:50:31.695861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.695824 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df58d996-eafc-4968-afaf-b6e6a42c594f-webhook-cert\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.695861 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.695862 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k956z\" (UniqueName: \"kubernetes.io/projected/df58d996-eafc-4968-afaf-b6e6a42c594f-kube-api-access-k956z\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.696086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.695953 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df58d996-eafc-4968-afaf-b6e6a42c594f-metrics-certs\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.797020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.796977 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df58d996-eafc-4968-afaf-b6e6a42c594f-webhook-cert\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.797020 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.797019 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k956z\" (UniqueName: \"kubernetes.io/projected/df58d996-eafc-4968-afaf-b6e6a42c594f-kube-api-access-k956z\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.797290 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.797060 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df58d996-eafc-4968-afaf-b6e6a42c594f-metrics-certs\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.799828 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.799794 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df58d996-eafc-4968-afaf-b6e6a42c594f-webhook-cert\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.799961 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.799917 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df58d996-eafc-4968-afaf-b6e6a42c594f-metrics-certs\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.810175 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.810147 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k956z\" (UniqueName: \"kubernetes.io/projected/df58d996-eafc-4968-afaf-b6e6a42c594f-kube-api-access-k956z\") pod \"maas-controller-587977945b-p7s6j\" (UID: \"df58d996-eafc-4968-afaf-b6e6a42c594f\") " pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:31.904359 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:31.904265 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:32.043752 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:32.043717 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2742ecf-252d-419f-98e6-35b3f9e37e96" path="/var/lib/kubelet/pods/a2742ecf-252d-419f-98e6-35b3f9e37e96/volumes" Nov 02 01:50:32.186271 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:32.186242 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587977945b-p7s6j"] Nov 02 01:50:32.187932 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:50:32.187900 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf58d996_eafc_4968_afaf_b6e6a42c594f.slice/crio-0965f7b959f091175953b4804ed8b36e8054c2fca2470fa8c92a18230a901072 WatchSource:0}: Error finding container 0965f7b959f091175953b4804ed8b36e8054c2fca2470fa8c92a18230a901072: Status 404 returned error can't find the container with id 0965f7b959f091175953b4804ed8b36e8054c2fca2470fa8c92a18230a901072 Nov 02 01:50:32.277738 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:32.277703 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-p7s6j" event={"ID":"df58d996-eafc-4968-afaf-b6e6a42c594f","Type":"ContainerStarted","Data":"0965f7b959f091175953b4804ed8b36e8054c2fca2470fa8c92a18230a901072"} Nov 02 01:50:33.286163 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:33.286105 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-p7s6j" event={"ID":"df58d996-eafc-4968-afaf-b6e6a42c594f","Type":"ContainerStarted","Data":"7b1c20b2e2a6a593605fc0068f2b0c125a20d3846e4abe5accdb2b5d9896a4ea"} Nov 02 01:50:33.286671 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:33.286235 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:33.308546 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:33.308487 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-587977945b-p7s6j" podStartSLOduration=1.552131432 podStartE2EDuration="2.30847191s" totalImagesPullingTime="756.340478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:33.305691274 +0000 UTC m=+2133.411480191" watchObservedRunningTime="2026-11-02 01:50:33.30847191 +0000 UTC m=+2133.414260802" Nov 02 01:50:44.295093 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:50:44.295061 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587977945b-p7s6j" Nov 02 01:50:57.750807 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:50:57.750766 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:51:06.443934 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:51:06.443891 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:51:22.309209 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:51:22.309171 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:51:26.158858 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:51:26.158795 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:51:27.751310 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:51:27.751274 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:51:41.007061 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:51:41.006976 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:51:54.042956 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:51:54.042923 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:51:57.750051 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:51:57.750014 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:52:18.758056 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:18.758020 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:52:27.749798 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:52:27.749758 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:52:29.006696 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:29.006663 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:52:32.772403 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:32.772351 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:52:33.160387 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:33.160282 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:52:33.160698 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:33.160665 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:52:49.615086 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:49.615052 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:52:53.534408 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:52:53.534372 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:52:57.750086 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:52:57.750046 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:53:07.742379 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:07.742347 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:53:12.861207 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:12.856020 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:53:26.605678 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:26.605643 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:53:27.752922 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:53:27.752878 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:53:30.447246 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:30.447205 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:53:44.314717 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:44.314681 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:53:55.018548 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:55.018514 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:53:57.818143 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:53:57.815794 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:53:59.139759 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:53:59.139710 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:54:20.212136 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:20.212088 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:54:27.750934 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:54:27.750898 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:54:30.026346 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:30.026310 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb"] Nov 02 01:54:30.029284 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:30.029258 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef767fd1-6xssb"] Nov 02 01:54:30.043174 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:30.043146 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27359d1d-4991-417c-b1c3-33a016b58101" path="/var/lib/kubelet/pods/27359d1d-4991-417c-b1c3-33a016b58101/volumes" Nov 02 01:54:35.023980 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:35.023936 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv"] Nov 02 01:54:35.029996 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:35.029962 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e073df1-896sv"] Nov 02 01:54:36.043648 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:36.043611 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558b782a-4d28-4e2e-8664-1c1f4a02e4dc" path="/var/lib/kubelet/pods/558b782a-4d28-4e2e-8664-1c1f4a02e4dc/volumes" Nov 02 01:54:38.353026 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:38.352982 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:54:48.535023 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:54:48.534934 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:54:57.750478 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:54:57.750439 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:55:04.510486 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:55:04.510453 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:55:08.625543 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:55:08.625506 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:55:19.343338 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:55:19.343307 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:55:27.762704 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:55:27.762666 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:55:30.751903 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:55:30.751851 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:55:45.424462 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:55:45.424423 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:55:57.751474 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:55:57.751435 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:56:03.553305 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:03.553269 4552 scope.go:122] "RemoveContainer" containerID="3f8bf205f26c3695c6ecce523dacc1272e15938d316df0142c3f21be2b6ea7ee" Nov 02 01:56:03.562970 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:03.562947 4552 scope.go:122] "RemoveContainer" containerID="c276c407b7128fe6d959ec0bf7bece907c04bea6569cf446c5886c6087ccb6bd" Nov 02 01:56:09.713172 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:09.713059 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:56:21.032935 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:21.032895 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:56:27.751177 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:56:27.751138 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:56:38.324150 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:38.324098 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:56:42.504025 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:42.503987 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:56:46.094785 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.094754 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg"] Nov 02 01:56:46.095277 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.094968 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" podUID="bc610634-845d-4214-96b5-a41d58c809f0" containerName="manager" containerID="cri-o://7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a" gracePeriod=10 Nov 02 01:56:46.356591 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.356532 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:56:46.386563 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.386531 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl\" (UniqueName: \"kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl\") pod \"bc610634-845d-4214-96b5-a41d58c809f0\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " Nov 02 01:56:46.386755 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.386612 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume\") pod \"bc610634-845d-4214-96b5-a41d58c809f0\" (UID: \"bc610634-845d-4214-96b5-a41d58c809f0\") " Nov 02 01:56:46.387494 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.387424 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume" pod "bc610634-845d-4214-96b5-a41d58c809f0" (UID: "bc610634-845d-4214-96b5-a41d58c809f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:56:46.390089 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.390061 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl" pod "bc610634-845d-4214-96b5-a41d58c809f0" (UID: "bc610634-845d-4214-96b5-a41d58c809f0"). InnerVolumeSpecName "kube-api-access-ms4zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:56:46.487921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.487889 4552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc610634-845d-4214-96b5-a41d58c809f0-extensions-socket-volume\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:46.487921 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.487915 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4zl\" (UniqueName: \"kubernetes.io/projected/bc610634-845d-4214-96b5-a41d58c809f0-kube-api-access-ms4zl\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:46.879875 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.879835 4552 generic.go:356] "Generic (PLEG): container finished" podID="bc610634-845d-4214-96b5-a41d58c809f0" containerID="7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a" exitCode=0 Nov 02 01:56:46.880100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.879900 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" Nov 02 01:56:46.880100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.879928 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" event={"ID":"bc610634-845d-4214-96b5-a41d58c809f0","Type":"ContainerDied","Data":"7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a"} Nov 02 01:56:46.880100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.880018 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg" event={"ID":"bc610634-845d-4214-96b5-a41d58c809f0","Type":"ContainerDied","Data":"8b6c9fb96a7710999ac9d7919e65034326127b3dbf0b96251fce6cf4ea59497d"} Nov 02 01:56:46.880100 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.880049 4552 scope.go:122] "RemoveContainer" containerID="7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a" Nov 02 01:56:46.891780 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.891760 4552 scope.go:122] "RemoveContainer" containerID="7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a" Nov 02 01:56:46.892052 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:56:46.892032 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a\": container with ID starting with 7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a not found: ID does not exist" containerID="7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a" Nov 02 01:56:46.892249 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.892064 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a"} err="failed to get container status \"7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a\": rpc error: code = NotFound desc = could not find container \"7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a\": container with ID starting with 7c655cc95c67a9cb127b3832593689acc17bb041de0ae2f36f506bd7d3e8a42a not found: ID does not exist" Nov 02 01:56:46.904812 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.904788 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg"] Nov 02 01:56:46.910088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:46.910065 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qgwgg"] Nov 02 01:56:48.042946 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:56:48.042910 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc610634-845d-4214-96b5-a41d58c809f0" path="/var/lib/kubelet/pods/bc610634-845d-4214-96b5-a41d58c809f0/volumes" Nov 02 01:56:57.749635 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:56:57.749590 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:57:27.749378 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:57:27.749337 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:57:47.138909 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.138832 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6"] Nov 02 01:57:47.139345 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.139331 4552 cpu_manager.go:414] "containerMap: removing container" podUID="bc610634-845d-4214-96b5-a41d58c809f0" containerName="manager" Nov 02 01:57:47.139390 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.139346 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc610634-845d-4214-96b5-a41d58c809f0" containerName="manager" Nov 02 01:57:47.139442 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.139434 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc610634-845d-4214-96b5-a41d58c809f0" containerName="manager" Nov 02 01:57:47.142695 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.142678 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.146600 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.146566 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n8fpr\"" Nov 02 01:57:47.156898 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.156871 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6"] Nov 02 01:57:47.316781 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.316748 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znw2x\" (UniqueName: \"kubernetes.io/projected/ad4a123b-5263-4bc8-bccf-619b69e57291-kube-api-access-znw2x\") pod \"kuadrant-operator-controller-manager-678d87766f-ff6h6\" (UID: \"ad4a123b-5263-4bc8-bccf-619b69e57291\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.316967 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.316788 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad4a123b-5263-4bc8-bccf-619b69e57291-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ff6h6\" (UID: \"ad4a123b-5263-4bc8-bccf-619b69e57291\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.417772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.417685 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znw2x\" (UniqueName: \"kubernetes.io/projected/ad4a123b-5263-4bc8-bccf-619b69e57291-kube-api-access-znw2x\") pod \"kuadrant-operator-controller-manager-678d87766f-ff6h6\" (UID: \"ad4a123b-5263-4bc8-bccf-619b69e57291\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.417772 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.417726 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad4a123b-5263-4bc8-bccf-619b69e57291-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ff6h6\" (UID: \"ad4a123b-5263-4bc8-bccf-619b69e57291\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.418088 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.418064 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad4a123b-5263-4bc8-bccf-619b69e57291-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ff6h6\" (UID: \"ad4a123b-5263-4bc8-bccf-619b69e57291\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.432854 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.432825 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znw2x\" (UniqueName: \"kubernetes.io/projected/ad4a123b-5263-4bc8-bccf-619b69e57291-kube-api-access-znw2x\") pod \"kuadrant-operator-controller-manager-678d87766f-ff6h6\" (UID: \"ad4a123b-5263-4bc8-bccf-619b69e57291\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.457906 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.457874 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:47.746001 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:47.745978 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6"] Nov 02 01:57:47.749157 ip-10-0-137-1 kubenswrapper[4552]: W1102 01:57:47.749101 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4a123b_5263_4bc8_bccf_619b69e57291.slice/crio-bac4b5e9ccaa32310e2a619841a9987fc0887ce3e91ca1ba6df654f2ccac936e WatchSource:0}: Error finding container bac4b5e9ccaa32310e2a619841a9987fc0887ce3e91ca1ba6df654f2ccac936e: Status 404 returned error can't find the container with id bac4b5e9ccaa32310e2a619841a9987fc0887ce3e91ca1ba6df654f2ccac936e Nov 02 01:57:48.236531 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:48.236495 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" event={"ID":"ad4a123b-5263-4bc8-bccf-619b69e57291","Type":"ContainerStarted","Data":"5096f8d4060145cc7aaf3282ae93927ece96e14054c299f3a8b59198801abd5d"} Nov 02 01:57:48.237029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:48.236553 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" event={"ID":"ad4a123b-5263-4bc8-bccf-619b69e57291","Type":"ContainerStarted","Data":"bac4b5e9ccaa32310e2a619841a9987fc0887ce3e91ca1ba6df654f2ccac936e"} Nov 02 01:57:48.237029 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:48.236670 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:57:48.260799 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:48.260746 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" podStartSLOduration=1.260728412 podStartE2EDuration="1.260728412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:48.258422329 +0000 UTC m=+2568.364211244" watchObservedRunningTime="2026-11-02 01:57:48.260728412 +0000 UTC m=+2568.366517327" Nov 02 01:57:57.750875 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:57:57.750833 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:57:59.245166 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:57:59.245135 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ff6h6" Nov 02 01:58:04.851350 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:58:04.851312 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:58:19.910821 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:58:19.910777 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:58:23.715202 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:58:23.715161 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:58:27.749542 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:58:27.749501 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:58:38.733539 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:58:38.733499 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:58:47.042080 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:58:47.042036 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:58:56.845478 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:58:56.845438 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:58:57.751795 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:58:57.751761 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:59:12.749517 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:59:12.749415 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:59:27.750803 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:59:27.750758 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 01:59:36.343560 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:59:36.343517 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:59:48.724526 ip-10-0-137-1 kubenswrapper[4552]: I1102 01:59:48.724493 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 01:59:57.749538 ip-10-0-137-1 kubenswrapper[4552]: E1102 01:59:57.749498 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:00:00.325497 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:00.325464 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:00:03.166143 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:03.164251 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p24rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:03.166143 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:03.164356 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p24rm" podUID="a36874ab-fccd-45b7-9497-668a0fd1c983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:00:16.045356 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:16.045320 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:00:19.234283 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:19.234236 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:19.234893 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:19.234316 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:00:27.751955 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:00:27.751910 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:00:32.301435 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:32.301399 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:00:43.952303 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:43.952214 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:00:49.790357 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:49.790303 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:00:57.749657 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:00:57.749620 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:00:59.155011 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:00:59.154968 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:01:03.633143 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:03.632812 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:01:04.232143 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:04.229402 4552 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4nkxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:01:04.232143 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:04.229525 4552 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4nkxc" podUID="a9f2a834-c86c-4af6-bbde-17ba9e356e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:01:19.508391 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:19.508352 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:01:27.749709 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:01:27.749671 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:01:34.442067 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:34.442026 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:01:37.074686 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:37.074650 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-427qz"] Nov 02 02:01:37.075155 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:37.074868 4552 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" podUID="fb1d7096-00de-46aa-969a-72a24fb54adb" containerName="maas-api" containerID="cri-o://5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932" gracePeriod=30 Nov 02 02:01:38.358053 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.358030 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 02:01:38.395760 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.395726 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls\") pod \"fb1d7096-00de-46aa-969a-72a24fb54adb\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " Nov 02 02:01:38.395760 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.395763 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs\") pod \"fb1d7096-00de-46aa-969a-72a24fb54adb\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " Nov 02 02:01:38.395964 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.395893 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca\") pod \"fb1d7096-00de-46aa-969a-72a24fb54adb\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " Nov 02 02:01:38.395964 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.395941 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7\" (UniqueName: \"kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7\") pod \"fb1d7096-00de-46aa-969a-72a24fb54adb\" (UID: \"fb1d7096-00de-46aa-969a-72a24fb54adb\") " Nov 02 02:01:38.396316 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.396287 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca" pod "fb1d7096-00de-46aa-969a-72a24fb54adb" (UID: "fb1d7096-00de-46aa-969a-72a24fb54adb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:01:38.398203 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.398170 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7" pod "fb1d7096-00de-46aa-969a-72a24fb54adb" (UID: "fb1d7096-00de-46aa-969a-72a24fb54adb"). InnerVolumeSpecName "kube-api-access-pfkx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:38.398333 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.398205 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls" pod "fb1d7096-00de-46aa-969a-72a24fb54adb" (UID: "fb1d7096-00de-46aa-969a-72a24fb54adb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:38.398513 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.398496 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs" pod "fb1d7096-00de-46aa-969a-72a24fb54adb" (UID: "fb1d7096-00de-46aa-969a-72a24fb54adb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:38.497691 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.497659 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-maas-api-tls\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:38.497691 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.497690 4552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1d7096-00de-46aa-969a-72a24fb54adb-metrics-certs\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:38.497691 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.497700 4552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1d7096-00de-46aa-969a-72a24fb54adb-openshift-service-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:38.497966 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.497708 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkx7\" (UniqueName: \"kubernetes.io/projected/fb1d7096-00de-46aa-969a-72a24fb54adb-kube-api-access-pfkx7\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:38.895979 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.895888 4552 generic.go:356] "Generic (PLEG): container finished" podID="fb1d7096-00de-46aa-969a-72a24fb54adb" containerID="5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932" exitCode=0 Nov 02 02:01:38.895979 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.895937 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" Nov 02 02:01:38.896216 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.895985 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" event={"ID":"fb1d7096-00de-46aa-969a-72a24fb54adb","Type":"ContainerDied","Data":"5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932"} Nov 02 02:01:38.896216 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.896040 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-427qz" event={"ID":"fb1d7096-00de-46aa-969a-72a24fb54adb","Type":"ContainerDied","Data":"526b2bc0f23a1e4682483bc5f2cf921e0846d195a37f30587d87f5d35b566335"} Nov 02 02:01:38.896216 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.896067 4552 scope.go:122] "RemoveContainer" containerID="5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932" Nov 02 02:01:38.907443 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.907420 4552 scope.go:122] "RemoveContainer" containerID="5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932" Nov 02 02:01:38.907757 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:01:38.907736 4552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932\": container with ID starting with 5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932 not found: ID does not exist" containerID="5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932" Nov 02 02:01:38.907850 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.907765 4552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932"} err="failed to get container status \"5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932\": rpc error: code = NotFound desc = could not find container \"5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932\": container with ID starting with 5dc89e6a660fb4024404440563c0a4a2aa42d0ff1871ede65bc57924549cd932 not found: ID does not exist" Nov 02 02:01:38.940305 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.940267 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-427qz"] Nov 02 02:01:38.940507 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:38.940343 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-427qz"] Nov 02 02:01:40.044393 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:40.044359 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1d7096-00de-46aa-969a-72a24fb54adb" path="/var/lib/kubelet/pods/fb1d7096-00de-46aa-969a-72a24fb54adb/volumes" Nov 02 02:01:46.101907 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.101873 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w"] Nov 02 02:01:46.102385 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.102371 4552 cpu_manager.go:414] "containerMap: removing container" podUID="fb1d7096-00de-46aa-969a-72a24fb54adb" containerName="maas-api" Nov 02 02:01:46.102430 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.102386 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1d7096-00de-46aa-969a-72a24fb54adb" containerName="maas-api" Nov 02 02:01:46.102485 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.102476 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1d7096-00de-46aa-969a-72a24fb54adb" containerName="maas-api" Nov 02 02:01:46.107102 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.107082 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.111034 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.111001 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:01:46.111184 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.111068 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 02:01:46.112022 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.112001 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:01:46.112138 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.112037 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 02:01:46.121189 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.121165 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w"] Nov 02 02:01:46.189234 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.189200 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4gn\" (UniqueName: \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.189422 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.189256 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.189422 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.189383 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.290831 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.290794 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4gn\" (UniqueName: \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.291017 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.290850 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.291017 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.290887 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.291559 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.291537 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.293431 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.293406 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.328555 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:01:46.328525 4552 projected.go:196] Error preparing data for projected volume kube-api-access-qb4gn for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 02:01:46.328700 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:01:46.328619 4552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn podName:73442a6e-a294-4cca-a5c7-2c80037ddd26 nodeName:}" failed. No retries permitted until 2026-11-02 02:01:46.828597891 +0000 UTC m=+2806.934386797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qb4gn" (UniqueName: "kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn") pod "maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" (UID: "73442a6e-a294-4cca-a5c7-2c80037ddd26") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 02:01:46.897129 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.897092 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4gn\" (UniqueName: \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:46.909911 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:46.909878 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4gn\" (UniqueName: \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\") pod \"maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:47.021212 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.021173 4552 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" secret="" err="secret \"maas-api-cleanup-dockercfg-kwf5g\" not found" Nov 02 02:01:47.021404 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.021240 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:47.339956 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.339875 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w"] Nov 02 02:01:47.349627 ip-10-0-137-1 kubenswrapper[4552]: W1102 02:01:47.349592 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73442a6e_a294_4cca_a5c7_2c80037ddd26.slice/crio-29efafc9144ae424e8b7ec39e4776d9ea19a6bcaacf01dfc76a9eeac56eab964 WatchSource:0}: Error finding container 29efafc9144ae424e8b7ec39e4776d9ea19a6bcaacf01dfc76a9eeac56eab964: Status 404 returned error can't find the container with id 29efafc9144ae424e8b7ec39e4776d9ea19a6bcaacf01dfc76a9eeac56eab964 Nov 02 02:01:47.969922 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.969888 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" event={"ID":"73442a6e-a294-4cca-a5c7-2c80037ddd26","Type":"ContainerStarted","Data":"5e89f759dc24485eb63487262620746cb5c39a1ec84e3dda7a393cabffb108db"} Nov 02 02:01:47.970091 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.969943 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" event={"ID":"73442a6e-a294-4cca-a5c7-2c80037ddd26","Type":"ContainerStarted","Data":"29efafc9144ae424e8b7ec39e4776d9ea19a6bcaacf01dfc76a9eeac56eab964"} Nov 02 02:01:47.970215 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.970201 4552 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" secret="" err="secret \"maas-api-cleanup-dockercfg-kwf5g\" not found" Nov 02 02:01:47.987066 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:47.987015 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" podStartSLOduration=1.986997518 podStartE2EDuration="1.986997518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:47.986507973 +0000 UTC m=+2808.092296887" watchObservedRunningTime="2026-11-02 02:01:47.986997518 +0000 UTC m=+2808.092786433" Nov 02 02:01:48.977501 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:48.977466 4552 generic.go:356] "Generic (PLEG): container finished" podID="73442a6e-a294-4cca-a5c7-2c80037ddd26" containerID="5e89f759dc24485eb63487262620746cb5c39a1ec84e3dda7a393cabffb108db" exitCode=0 Nov 02 02:01:48.977929 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:48.977547 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" event={"ID":"73442a6e-a294-4cca-a5c7-2c80037ddd26","Type":"ContainerDied","Data":"5e89f759dc24485eb63487262620746cb5c39a1ec84e3dda7a393cabffb108db"} Nov 02 02:01:50.135291 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.135261 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:50.237651 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.237560 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca\") pod \"73442a6e-a294-4cca-a5c7-2c80037ddd26\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " Nov 02 02:01:50.237825 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.237720 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\" (UniqueName: \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\") pod \"73442a6e-a294-4cca-a5c7-2c80037ddd26\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " Nov 02 02:01:50.237825 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.237797 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca\") pod \"73442a6e-a294-4cca-a5c7-2c80037ddd26\" (UID: \"73442a6e-a294-4cca-a5c7-2c80037ddd26\") " Nov 02 02:01:50.237979 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.237953 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca" pod "73442a6e-a294-4cca-a5c7-2c80037ddd26" (UID: "73442a6e-a294-4cca-a5c7-2c80037ddd26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:01:50.238201 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.238167 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:50.239986 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.239964 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn" pod "73442a6e-a294-4cca-a5c7-2c80037ddd26" (UID: "73442a6e-a294-4cca-a5c7-2c80037ddd26"). InnerVolumeSpecName "kube-api-access-qb4gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:50.240164 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.240145 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca" pod "73442a6e-a294-4cca-a5c7-2c80037ddd26" (UID: "73442a6e-a294-4cca-a5c7-2c80037ddd26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:50.339674 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.339637 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4gn\" (UniqueName: \"kubernetes.io/projected/73442a6e-a294-4cca-a5c7-2c80037ddd26-kube-api-access-qb4gn\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:50.339674 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.339669 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73442a6e-a294-4cca-a5c7-2c80037ddd26-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:01:50.991964 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.991937 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" Nov 02 02:01:50.991964 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.991936 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w" event={"ID":"73442a6e-a294-4cca-a5c7-2c80037ddd26","Type":"ContainerDied","Data":"29efafc9144ae424e8b7ec39e4776d9ea19a6bcaacf01dfc76a9eeac56eab964"} Nov 02 02:01:50.992226 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:50.992037 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29efafc9144ae424e8b7ec39e4776d9ea19a6bcaacf01dfc76a9eeac56eab964" Nov 02 02:01:57.795568 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:01:57.795526 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:01:57.860729 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:57.860696 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w"] Nov 02 02:01:57.863401 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:57.863374 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad5ce34c-6cx6w"] Nov 02 02:01:58.043906 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:58.043873 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73442a6e-a294-4cca-a5c7-2c80037ddd26" path="/var/lib/kubelet/pods/73442a6e-a294-4cca-a5c7-2c80037ddd26/volumes" Nov 02 02:01:59.146884 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:01:59.146848 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:02:11.418235 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:02:11.418145 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:02:27.750002 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:02:27.749897 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:02:57.750816 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:02:57.750773 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:03:27.751609 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:03:27.751569 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:03:42.716143 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.716051 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj"] Nov 02 02:03:42.716652 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.716634 4552 cpu_manager.go:414] "containerMap: removing container" podUID="73442a6e-a294-4cca-a5c7-2c80037ddd26" containerName="revoke-keys" Nov 02 02:03:42.716699 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.716656 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73442a6e-a294-4cca-a5c7-2c80037ddd26" containerName="revoke-keys" Nov 02 02:03:42.716763 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.716754 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="73442a6e-a294-4cca-a5c7-2c80037ddd26" containerName="revoke-keys" Nov 02 02:03:42.719512 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.719495 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.726910 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.726885 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:03:42.727022 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.726955 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 02:03:42.727022 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.726999 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w5xl4\"" Nov 02 02:03:42.727022 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.727002 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:03:42.730873 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.730842 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj"] Nov 02 02:03:42.826372 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.826331 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.826565 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.826455 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.826565 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.826486 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjm88\" (UniqueName: \"kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.927747 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.927707 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.928004 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.927834 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.928004 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.927871 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjm88\" (UniqueName: \"kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.928637 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.928613 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.930468 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.930442 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:42.940272 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:42.940239 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjm88\" (UniqueName: \"kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:43.034480 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:43.034391 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:43.315898 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:43.315864 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj"] Nov 02 02:03:43.317719 ip-10-0-137-1 kubenswrapper[4552]: W1102 02:03:43.317684 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583d4d85_f4de_49a5_a0de_822461e77622.slice/crio-ab6dd11214f80fce852cd27d08663abb7867dc79caa3ab529f299cc36e030806 WatchSource:0}: Error finding container ab6dd11214f80fce852cd27d08663abb7867dc79caa3ab529f299cc36e030806: Status 404 returned error can't find the container with id ab6dd11214f80fce852cd27d08663abb7867dc79caa3ab529f299cc36e030806 Nov 02 02:03:43.728825 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:43.728784 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" event={"ID":"583d4d85-f4de-49a5-a0de-822461e77622","Type":"ContainerStarted","Data":"ab6dd11214f80fce852cd27d08663abb7867dc79caa3ab529f299cc36e030806"} Nov 02 02:03:44.736930 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:44.736894 4552 generic.go:356] "Generic (PLEG): container finished" podID="583d4d85-f4de-49a5-a0de-822461e77622" containerID="b62e6439c9ce708ff837e81b7a4099033465927b7b2457cb4f6c834636beb426" exitCode=0 Nov 02 02:03:44.737460 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:44.736983 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" event={"ID":"583d4d85-f4de-49a5-a0de-822461e77622","Type":"ContainerDied","Data":"b62e6439c9ce708ff837e81b7a4099033465927b7b2457cb4f6c834636beb426"} Nov 02 02:03:45.902285 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:45.902257 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:46.064512 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.064420 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88\" (UniqueName: \"kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88\") pod \"583d4d85-f4de-49a5-a0de-822461e77622\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " Nov 02 02:03:46.064512 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.064500 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca\") pod \"583d4d85-f4de-49a5-a0de-822461e77622\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " Nov 02 02:03:46.064769 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.064589 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca\") pod \"583d4d85-f4de-49a5-a0de-822461e77622\" (UID: \"583d4d85-f4de-49a5-a0de-822461e77622\") " Nov 02 02:03:46.064868 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.064842 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca" pod "583d4d85-f4de-49a5-a0de-822461e77622" (UID: "583d4d85-f4de-49a5-a0de-822461e77622"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:03:46.066862 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.066840 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca" pod "583d4d85-f4de-49a5-a0de-822461e77622" (UID: "583d4d85-f4de-49a5-a0de-822461e77622"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:03:46.066935 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.066861 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88" pod "583d4d85-f4de-49a5-a0de-822461e77622" (UID: "583d4d85-f4de-49a5-a0de-822461e77622"). InnerVolumeSpecName "kube-api-access-tjm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:03:46.165829 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.165789 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:03:46.165829 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.165823 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjm88\" (UniqueName: \"kubernetes.io/projected/583d4d85-f4de-49a5-a0de-822461e77622-kube-api-access-tjm88\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:03:46.165829 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.165837 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d4d85-f4de-49a5-a0de-822461e77622-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:03:46.750849 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.750810 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" event={"ID":"583d4d85-f4de-49a5-a0de-822461e77622","Type":"ContainerDied","Data":"ab6dd11214f80fce852cd27d08663abb7867dc79caa3ab529f299cc36e030806"} Nov 02 02:03:46.750849 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.750843 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj" Nov 02 02:03:46.751071 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:46.750872 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab6dd11214f80fce852cd27d08663abb7867dc79caa3ab529f299cc36e030806" Nov 02 02:03:57.750973 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:03:57.750936 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:03:58.164311 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:58.164226 4552 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj"] Nov 02 02:03:58.166873 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:58.166837 4552 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b12eb5-n6ksj"] Nov 02 02:03:59.632758 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:03:59.632720 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:04:00.043964 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:00.043918 4552 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583d4d85-f4de-49a5-a0de-822461e77622" path="/var/lib/kubelet/pods/583d4d85-f4de-49a5-a0de-822461e77622/volumes" Nov 02 02:04:07.127423 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:07.127386 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:04:12.414601 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:12.414559 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:04:16.740585 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.740551 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn"] Nov 02 02:04:16.741075 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.741058 4552 cpu_manager.go:414] "containerMap: removing container" podUID="583d4d85-f4de-49a5-a0de-822461e77622" containerName="revoke-keys" Nov 02 02:04:16.741156 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.741078 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583d4d85-f4de-49a5-a0de-822461e77622" containerName="revoke-keys" Nov 02 02:04:16.741197 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.741186 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="583d4d85-f4de-49a5-a0de-822461e77622" containerName="revoke-keys" Nov 02 02:04:16.745693 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.745676 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.750011 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.749988 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:04:16.751185 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.751162 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w5xl4\"" Nov 02 02:04:16.751185 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.751173 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:04:16.751673 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.751645 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 02:04:16.754889 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.754869 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn"] Nov 02 02:04:16.846387 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.846354 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62z9n\" (UniqueName: \"kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.846574 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.846412 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.846574 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.846527 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.947381 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.947343 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.947572 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.947446 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62z9n\" (UniqueName: \"kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.947572 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.947497 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.948073 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.948038 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.950079 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.950051 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:16.959765 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:16.959739 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62z9n\" (UniqueName: \"kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n\") pod \"maas-api-revoke-sub-f03629db-mg2sn\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:17.061172 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:17.061077 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:17.338221 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:17.338189 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn"] Nov 02 02:04:17.339735 ip-10-0-137-1 kubenswrapper[4552]: W1102 02:04:17.339709 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854ad8b9_fe77_43a8_a704_84303bb64b6d.slice/crio-cf9cd9de8ccd1a37c12f9707a9c448f3fbc43d18ee6f3b2194089ce0a321743f WatchSource:0}: Error finding container cf9cd9de8ccd1a37c12f9707a9c448f3fbc43d18ee6f3b2194089ce0a321743f: Status 404 returned error can't find the container with id cf9cd9de8ccd1a37c12f9707a9c448f3fbc43d18ee6f3b2194089ce0a321743f Nov 02 02:04:18.012303 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:18.012265 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" event={"ID":"854ad8b9-fe77-43a8-a704-84303bb64b6d","Type":"ContainerStarted","Data":"1cdcf07308000dc37942dc25d891ffec03a14d28b16c587c26b6c3ea17c26a6f"} Nov 02 02:04:18.012754 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:18.012330 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" event={"ID":"854ad8b9-fe77-43a8-a704-84303bb64b6d","Type":"ContainerStarted","Data":"cf9cd9de8ccd1a37c12f9707a9c448f3fbc43d18ee6f3b2194089ce0a321743f"} Nov 02 02:04:18.031841 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:18.031792 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" podStartSLOduration=2.03177772 podStartE2EDuration="2.03177772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:18.02952313 +0000 UTC m=+2958.135312044" watchObservedRunningTime="2026-11-02 02:04:18.03177772 +0000 UTC m=+2958.137566637" Nov 02 02:04:19.019538 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:19.019504 4552 generic.go:356] "Generic (PLEG): container finished" podID="854ad8b9-fe77-43a8-a704-84303bb64b6d" containerID="1cdcf07308000dc37942dc25d891ffec03a14d28b16c587c26b6c3ea17c26a6f" exitCode=0 Nov 02 02:04:19.019955 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:19.019602 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" event={"ID":"854ad8b9-fe77-43a8-a704-84303bb64b6d","Type":"ContainerDied","Data":"1cdcf07308000dc37942dc25d891ffec03a14d28b16c587c26b6c3ea17c26a6f"} Nov 02 02:04:20.172766 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.172736 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:20.280072 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.279990 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca\") pod \"854ad8b9-fe77-43a8-a704-84303bb64b6d\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " Nov 02 02:04:20.280258 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.280100 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca\") pod \"854ad8b9-fe77-43a8-a704-84303bb64b6d\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " Nov 02 02:04:20.280258 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.280208 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n\" (UniqueName: \"kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n\") pod \"854ad8b9-fe77-43a8-a704-84303bb64b6d\" (UID: \"854ad8b9-fe77-43a8-a704-84303bb64b6d\") " Nov 02 02:04:20.280371 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.280348 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca" pod "854ad8b9-fe77-43a8-a704-84303bb64b6d" (UID: "854ad8b9-fe77-43a8-a704-84303bb64b6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:20.280561 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.280542 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:20.282507 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.282478 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca" pod "854ad8b9-fe77-43a8-a704-84303bb64b6d" (UID: "854ad8b9-fe77-43a8-a704-84303bb64b6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:20.282618 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.282549 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n" pod "854ad8b9-fe77-43a8-a704-84303bb64b6d" (UID: "854ad8b9-fe77-43a8-a704-84303bb64b6d"). InnerVolumeSpecName "kube-api-access-62z9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:20.381449 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.381414 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854ad8b9-fe77-43a8-a704-84303bb64b6d-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:20.381449 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:20.381445 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62z9n\" (UniqueName: \"kubernetes.io/projected/854ad8b9-fe77-43a8-a704-84303bb64b6d-kube-api-access-62z9n\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:21.032441 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:21.032409 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" Nov 02 02:04:21.032631 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:21.032452 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mg2sn" event={"ID":"854ad8b9-fe77-43a8-a704-84303bb64b6d","Type":"ContainerDied","Data":"cf9cd9de8ccd1a37c12f9707a9c448f3fbc43d18ee6f3b2194089ce0a321743f"} Nov 02 02:04:21.032631 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:21.032510 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf9cd9de8ccd1a37c12f9707a9c448f3fbc43d18ee6f3b2194089ce0a321743f" Nov 02 02:04:27.651099 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:27.651063 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:04:27.750046 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:04:27.750004 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:04:32.695930 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.695891 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt"] Nov 02 02:04:32.696663 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.696642 4552 cpu_manager.go:414] "containerMap: removing container" podUID="854ad8b9-fe77-43a8-a704-84303bb64b6d" containerName="revoke-keys" Nov 02 02:04:32.696749 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.696666 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854ad8b9-fe77-43a8-a704-84303bb64b6d" containerName="revoke-keys" Nov 02 02:04:32.696839 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.696826 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="854ad8b9-fe77-43a8-a704-84303bb64b6d" containerName="revoke-keys" Nov 02 02:04:32.700247 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.700226 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.705989 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.705961 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:04:32.706166 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.706049 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:04:32.706610 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.706589 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 02:04:32.706716 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.706681 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w5xl4\"" Nov 02 02:04:32.708635 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.708615 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt"] Nov 02 02:04:32.740997 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.740971 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cc7\" (UniqueName: \"kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.741195 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.741080 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.741195 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.741133 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.841777 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.841733 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cc7\" (UniqueName: \"kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.841983 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.841836 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.841983 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.841875 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.842528 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.842509 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.844410 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.844380 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:32.854568 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:32.854545 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cc7\" (UniqueName: \"kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7\") pod \"maas-api-revoke-sub-0165d7d2-xjgbt\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:33.016224 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:33.016195 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:33.295940 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:33.295913 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt"] Nov 02 02:04:33.297256 ip-10-0-137-1 kubenswrapper[4552]: W1102 02:04:33.297224 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c41f180_0387_4bd0_9329_5407c5aef02f.slice/crio-63afbfcb3d20384512b56d1e6a4bc91b70a61e84c0e33bf124000604107b088e WatchSource:0}: Error finding container 63afbfcb3d20384512b56d1e6a4bc91b70a61e84c0e33bf124000604107b088e: Status 404 returned error can't find the container with id 63afbfcb3d20384512b56d1e6a4bc91b70a61e84c0e33bf124000604107b088e Nov 02 02:04:34.139248 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:34.139214 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" event={"ID":"6c41f180-0387-4bd0-9329-5407c5aef02f","Type":"ContainerStarted","Data":"f3c236c6a0aec6409062a08672963d800a1d0ac211c56594d9ae1d26d9f109c9"} Nov 02 02:04:34.139606 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:34.139270 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" event={"ID":"6c41f180-0387-4bd0-9329-5407c5aef02f","Type":"ContainerStarted","Data":"63afbfcb3d20384512b56d1e6a4bc91b70a61e84c0e33bf124000604107b088e"} Nov 02 02:04:34.159151 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:34.159090 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" podStartSLOduration=2.159077078 podStartE2EDuration="2.159077078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:34.156872549 +0000 UTC m=+2974.262661464" watchObservedRunningTime="2026-11-02 02:04:34.159077078 +0000 UTC m=+2974.264866044" Nov 02 02:04:35.147109 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:35.147075 4552 generic.go:356] "Generic (PLEG): container finished" podID="6c41f180-0387-4bd0-9329-5407c5aef02f" containerID="f3c236c6a0aec6409062a08672963d800a1d0ac211c56594d9ae1d26d9f109c9" exitCode=0 Nov 02 02:04:35.147517 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:35.147173 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" event={"ID":"6c41f180-0387-4bd0-9329-5407c5aef02f","Type":"ContainerDied","Data":"f3c236c6a0aec6409062a08672963d800a1d0ac211c56594d9ae1d26d9f109c9"} Nov 02 02:04:36.300271 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.300248 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:36.369406 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.369376 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca\") pod \"6c41f180-0387-4bd0-9329-5407c5aef02f\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " Nov 02 02:04:36.369576 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.369512 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca\") pod \"6c41f180-0387-4bd0-9329-5407c5aef02f\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " Nov 02 02:04:36.369576 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.369540 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7\" (UniqueName: \"kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7\") pod \"6c41f180-0387-4bd0-9329-5407c5aef02f\" (UID: \"6c41f180-0387-4bd0-9329-5407c5aef02f\") " Nov 02 02:04:36.369905 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.369880 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca" pod "6c41f180-0387-4bd0-9329-5407c5aef02f" (UID: "6c41f180-0387-4bd0-9329-5407c5aef02f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:36.371619 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.371591 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca" pod "6c41f180-0387-4bd0-9329-5407c5aef02f" (UID: "6c41f180-0387-4bd0-9329-5407c5aef02f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:36.371862 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.371843 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7" pod "6c41f180-0387-4bd0-9329-5407c5aef02f" (UID: "6c41f180-0387-4bd0-9329-5407c5aef02f"). InnerVolumeSpecName "kube-api-access-n6cc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:36.471104 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.471079 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.471104 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.471105 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6cc7\" (UniqueName: \"kubernetes.io/projected/6c41f180-0387-4bd0-9329-5407c5aef02f-kube-api-access-n6cc7\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.471319 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:36.471140 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c41f180-0387-4bd0-9329-5407c5aef02f-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:37.159624 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:37.159594 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" Nov 02 02:04:37.159849 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:37.159630 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-xjgbt" event={"ID":"6c41f180-0387-4bd0-9329-5407c5aef02f","Type":"ContainerDied","Data":"63afbfcb3d20384512b56d1e6a4bc91b70a61e84c0e33bf124000604107b088e"} Nov 02 02:04:37.159849 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:37.159676 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63afbfcb3d20384512b56d1e6a4bc91b70a61e84c0e33bf124000604107b088e" Nov 02 02:04:43.402320 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.402283 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz"] Nov 02 02:04:43.402906 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.402888 4552 cpu_manager.go:414] "containerMap: removing container" podUID="6c41f180-0387-4bd0-9329-5407c5aef02f" containerName="revoke-keys" Nov 02 02:04:43.403048 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.402909 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c41f180-0387-4bd0-9329-5407c5aef02f" containerName="revoke-keys" Nov 02 02:04:43.403156 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.403048 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c41f180-0387-4bd0-9329-5407c5aef02f" containerName="revoke-keys" Nov 02 02:04:43.406410 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.406394 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.416146 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.412937 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 02:04:43.416146 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.413011 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:04:43.416146 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.413042 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:04:43.416146 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.413087 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w5xl4\"" Nov 02 02:04:43.417267 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.417246 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz"] Nov 02 02:04:43.444477 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.444433 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtkq\" (UniqueName: \"kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.444639 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.444514 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.444698 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.444669 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.545387 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.545356 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.545590 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.545480 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtkq\" (UniqueName: \"kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.545590 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.545532 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.546144 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.546110 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.548572 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.548537 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.558327 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.558300 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtkq\" (UniqueName: \"kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq\") pod \"maas-api-revoke-sub-6b19338f-7rrrz\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:43.722685 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:43.722646 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:44.016709 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:44.016670 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz"] Nov 02 02:04:44.016947 ip-10-0-137-1 kubenswrapper[4552]: W1102 02:04:44.016924 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f36c25_78ef_48bd_aae1_ac4ff42641a2.slice/crio-771ef22d329121b38598a4281b93022d40db16d896454c509ad06d3777b3ab86 WatchSource:0}: Error finding container 771ef22d329121b38598a4281b93022d40db16d896454c509ad06d3777b3ab86: Status 404 returned error can't find the container with id 771ef22d329121b38598a4281b93022d40db16d896454c509ad06d3777b3ab86 Nov 02 02:04:44.204179 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:44.204142 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" event={"ID":"70f36c25-78ef-48bd-aae1-ac4ff42641a2","Type":"ContainerStarted","Data":"771ef22d329121b38598a4281b93022d40db16d896454c509ad06d3777b3ab86"} Nov 02 02:04:45.211478 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:45.211390 4552 generic.go:356] "Generic (PLEG): container finished" podID="70f36c25-78ef-48bd-aae1-ac4ff42641a2" containerID="040cedd251e88841fcbba0d2e80093cb85e91cf0fd06da55db2b4c6a9eb630e8" exitCode=0 Nov 02 02:04:45.211478 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:45.211468 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" event={"ID":"70f36c25-78ef-48bd-aae1-ac4ff42641a2","Type":"ContainerDied","Data":"040cedd251e88841fcbba0d2e80093cb85e91cf0fd06da55db2b4c6a9eb630e8"} Nov 02 02:04:46.369173 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.369149 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:46.478392 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.478300 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca\") pod \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " Nov 02 02:04:46.478556 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.478408 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca\") pod \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " Nov 02 02:04:46.478556 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.478445 4552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq\" (UniqueName: \"kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq\") pod \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\" (UID: \"70f36c25-78ef-48bd-aae1-ac4ff42641a2\") " Nov 02 02:04:46.478762 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.478738 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca" pod "70f36c25-78ef-48bd-aae1-ac4ff42641a2" (UID: "70f36c25-78ef-48bd-aae1-ac4ff42641a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:46.480652 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.480619 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca" pod "70f36c25-78ef-48bd-aae1-ac4ff42641a2" (UID: "70f36c25-78ef-48bd-aae1-ac4ff42641a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:46.480652 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.480636 4552 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq" pod "70f36c25-78ef-48bd-aae1-ac4ff42641a2" (UID: "70f36c25-78ef-48bd-aae1-ac4ff42641a2"). InnerVolumeSpecName "kube-api-access-6qtkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:46.579668 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.579626 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:46.579668 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.579661 4552 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f36c25-78ef-48bd-aae1-ac4ff42641a2-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:46.579668 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:46.579674 4552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qtkq\" (UniqueName: \"kubernetes.io/projected/70f36c25-78ef-48bd-aae1-ac4ff42641a2-kube-api-access-6qtkq\") on node \"ip-10-0-137-1.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 02:04:47.225488 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:47.225455 4552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" Nov 02 02:04:47.225719 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:47.225483 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7rrrz" event={"ID":"70f36c25-78ef-48bd-aae1-ac4ff42641a2","Type":"ContainerDied","Data":"771ef22d329121b38598a4281b93022d40db16d896454c509ad06d3777b3ab86"} Nov 02 02:04:47.225719 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:47.225532 4552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="771ef22d329121b38598a4281b93022d40db16d896454c509ad06d3777b3ab86" Nov 02 02:04:54.229793 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:04:54.229749 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8z525"] Nov 02 02:04:57.749660 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:04:57.749619 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:05:06.489784 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:06.489748 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kwxtx_14d4c10d-b4c8-4973-ba7f-56b3e857d4b5/manager/0.log" Nov 02 02:05:06.764674 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:06.764599 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4xbm_5e507870-0bc0-4bb7-b0cb-381084e3a5c1/manager/0.log" Nov 02 02:05:07.328760 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:07.328727 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587977945b-p7s6j_df58d996-eafc-4968-afaf-b6e6a42c594f/manager/0.log" Nov 02 02:05:07.590232 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:07.590144 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s8xz5_7f972685-30f2-4bcc-9770-0c3a344d0dc7/server/0.log" Nov 02 02:05:08.157207 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:08.157176 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rx7hr_d8ed378a-6a38-423b-a3d2-9c2622a65d42/proxy/0.log" Nov 02 02:05:09.664351 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:09.664270 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-xjgbt_6c41f180-0387-4bd0-9329-5407c5aef02f/revoke-keys/0.log" Nov 02 02:05:10.375478 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:10.375445 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-7rrrz_70f36c25-78ef-48bd-aae1-ac4ff42641a2/revoke-keys/0.log" Nov 02 02:05:10.842303 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:10.842274 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-mg2sn_854ad8b9-fe77-43a8-a704-84303bb64b6d/revoke-keys/0.log" Nov 02 02:05:12.894188 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:12.894161 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-lbxvg_67d4d1aa-9743-4739-b659-94b11f451e2e/manager/0.log" Nov 02 02:05:13.080374 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:13.080346 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dsrsf_de8461ad-fdf0-462c-ad1f-e1dc0dd490be/manager/0.log" Nov 02 02:05:13.461179 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:13.461147 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ff6h6_ad4a123b-5263-4bc8-bccf-619b69e57291/manager/0.log" Nov 02 02:05:13.667398 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:13.667372 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8z525_acd9cb01-352d-4902-b532-7f12ae04448f/limitador/0.log" Nov 02 02:05:13.840819 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:13.840788 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6w6cf_4253a47b-9095-4961-aa62-e6a4e4fd513e/manager/0.log" Nov 02 02:05:14.605467 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:14.605432 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489g8hq9_f1472bbc-66ed-4e4e-830c-87290f40c66a/istio-proxy/0.log" Nov 02 02:05:14.791427 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:14.791395 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q47qx_dd923345-a8ad-45e5-8a8f-ab39df962b65/discovery/0.log" Nov 02 02:05:15.572860 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:15.572830 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/1.log" Nov 02 02:05:16.893262 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:16.893233 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-p24rm_a36874ab-fccd-45b7-9497-668a0fd1c983/istio-proxy/0.log" Nov 02 02:05:17.091924 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:17.091897 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-q856k_7f5f593a-2283-4295-acd1-909ae2384f98/istio-proxy/0.log" Nov 02 02:05:17.292313 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:17.292279 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9_3f3b5284-e92a-42aa-95b0-0fe67d121bd7/istio-proxy/0.log" Nov 02 02:05:17.655167 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:17.655084 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:05:18.345697 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:18.345667 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/0.log" Nov 02 02:05:18.912336 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:18.912299 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm_be4f2277-8eb3-42bb-8384-64bf743cb7aa/main/0.log" Nov 02 02:05:19.111109 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:19.111074 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7qsc7_5fe9f839-921c-433a-a857-9532cae45e2b/main/0.log" Nov 02 02:05:19.507040 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:19.507005 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kb25x_a31aa41f-6bdd-4401-b45b-8a7e863305ae/main/0.log" Nov 02 02:05:20.080922 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:20.080893 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th_d82568ce-5132-457b-8166-ec3b93c94297/main/0.log" Nov 02 02:05:27.750031 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:05:27.749995 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:05:30.777507 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:30.777476 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kwxtx_14d4c10d-b4c8-4973-ba7f-56b3e857d4b5/manager/0.log" Nov 02 02:05:30.972382 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:30.972357 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4xbm_5e507870-0bc0-4bb7-b0cb-381084e3a5c1/manager/0.log" Nov 02 02:05:31.400490 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:31.400460 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587977945b-p7s6j_df58d996-eafc-4968-afaf-b6e6a42c594f/manager/0.log" Nov 02 02:05:31.632158 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:31.632107 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s8xz5_7f972685-30f2-4bcc-9770-0c3a344d0dc7/server/0.log" Nov 02 02:05:32.038598 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:32.038565 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rx7hr_d8ed378a-6a38-423b-a3d2-9c2622a65d42/proxy/0.log" Nov 02 02:05:33.027646 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:33.027617 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-xjgbt_6c41f180-0387-4bd0-9329-5407c5aef02f/revoke-keys/0.log" Nov 02 02:05:33.563037 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:33.563009 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-7rrrz_70f36c25-78ef-48bd-aae1-ac4ff42641a2/revoke-keys/0.log" Nov 02 02:05:33.951424 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:33.951331 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-mg2sn_854ad8b9-fe77-43a8-a704-84303bb64b6d/revoke-keys/0.log" Nov 02 02:05:35.527497 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:35.527464 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-lbxvg_67d4d1aa-9743-4739-b659-94b11f451e2e/manager/0.log" Nov 02 02:05:35.709664 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:35.709631 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dsrsf_de8461ad-fdf0-462c-ad1f-e1dc0dd490be/manager/0.log" Nov 02 02:05:36.077429 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:36.077397 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ff6h6_ad4a123b-5263-4bc8-bccf-619b69e57291/manager/0.log" Nov 02 02:05:36.276860 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:36.276830 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8z525_acd9cb01-352d-4902-b532-7f12ae04448f/limitador/0.log" Nov 02 02:05:36.487873 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:36.487845 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6w6cf_4253a47b-9095-4961-aa62-e6a4e4fd513e/manager/0.log" Nov 02 02:05:37.064757 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:37.064727 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489g8hq9_f1472bbc-66ed-4e4e-830c-87290f40c66a/istio-proxy/0.log" Nov 02 02:05:37.248968 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:37.248941 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q47qx_dd923345-a8ad-45e5-8a8f-ab39df962b65/discovery/0.log" Nov 02 02:05:37.984778 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:37.984743 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/1.log" Nov 02 02:05:39.304501 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:39.304470 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-p24rm_a36874ab-fccd-45b7-9497-668a0fd1c983/istio-proxy/0.log" Nov 02 02:05:39.476240 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:39.476208 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-q856k_7f5f593a-2283-4295-acd1-909ae2384f98/istio-proxy/0.log" Nov 02 02:05:39.671710 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:39.671626 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9_3f3b5284-e92a-42aa-95b0-0fe67d121bd7/istio-proxy/0.log" Nov 02 02:05:40.036419 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:40.036391 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:05:40.703083 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:40.703048 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-s5nwm_be4f2277-8eb3-42bb-8384-64bf743cb7aa/main/0.log" Nov 02 02:05:40.880390 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:40.880356 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7qsc7_5fe9f839-921c-433a-a857-9532cae45e2b/main/0.log" Nov 02 02:05:41.245090 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:41.245062 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kb25x_a31aa41f-6bdd-4401-b45b-8a7e863305ae/main/0.log" Nov 02 02:05:41.798177 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:05:41.798142 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lc2th_d82568ce-5132-457b-8166-ec3b93c94297/main/0.log" Nov 02 02:05:57.749183 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:05:57.749111 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:06:19.637804 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:06:19.637768 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587977945b-p7s6j_df58d996-eafc-4968-afaf-b6e6a42c594f/manager/0.log" Nov 02 02:06:19.883721 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:06:19.883691 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587977945b-p7s6j_df58d996-eafc-4968-afaf-b6e6a42c594f/manager/0.log" Nov 02 02:06:20.574163 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:06:20.574102 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:06:20.748919 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:06:20.748889 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:06:27.750041 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:06:27.750004 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:06:57.750261 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:06:57.750181 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:07:07.442277 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:07.442250 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587977945b-p7s6j_df58d996-eafc-4968-afaf-b6e6a42c594f/manager/0.log" Nov 02 02:07:07.696671 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:07.696597 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587977945b-p7s6j_df58d996-eafc-4968-afaf-b6e6a42c594f/manager/0.log" Nov 02 02:07:08.396471 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:08.396443 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:07:08.573730 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:08.573701 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:07:12.966988 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:12.966954 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tqwk8_dc2693cb-eac1-4cd0-b76c-db35732e9635/global-pull-secret-syncer/0.log" Nov 02 02:07:13.189974 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:13.189944 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qjmzq_31cdfa2e-154d-443b-b5ef-7fbea8c1b3fa/konnectivity-agent/0.log" Nov 02 02:07:13.296169 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:13.296081 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-1.us-east-2.compute.internal_0100e2035cafa00853427683c7ebd01f/haproxy/0.log" Nov 02 02:07:20.172960 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:20.172932 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-lbxvg_67d4d1aa-9743-4739-b659-94b11f451e2e/manager/0.log" Nov 02 02:07:20.207616 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:20.207588 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dsrsf_de8461ad-fdf0-462c-ad1f-e1dc0dd490be/manager/0.log" Nov 02 02:07:20.299159 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:20.299102 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ff6h6_ad4a123b-5263-4bc8-bccf-619b69e57291/manager/0.log" Nov 02 02:07:20.322440 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:20.322409 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8z525_acd9cb01-352d-4902-b532-7f12ae04448f/limitador/0.log" Nov 02 02:07:20.396404 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:20.396369 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6w6cf_4253a47b-9095-4961-aa62-e6a4e4fd513e/manager/0.log" Nov 02 02:07:21.769222 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.769193 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/alertmanager/0.log" Nov 02 02:07:21.794804 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.794774 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/config-reloader/0.log" Nov 02 02:07:21.819023 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.818998 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/kube-rbac-proxy-web/0.log" Nov 02 02:07:21.849940 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.849906 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/kube-rbac-proxy/0.log" Nov 02 02:07:21.880800 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.880773 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/kube-rbac-proxy-metric/0.log" Nov 02 02:07:21.905618 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.905595 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/prom-label-proxy/0.log" Nov 02 02:07:21.936206 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:21.936180 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6e097c5-cc31-4829-888e-3837adf23172/init-config-reloader/0.log" Nov 02 02:07:22.026003 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.025924 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lq2pz_305e6b99-4656-459a-95a4-c98da237618f/kube-state-metrics/0.log" Nov 02 02:07:22.050220 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.050192 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lq2pz_305e6b99-4656-459a-95a4-c98da237618f/kube-rbac-proxy-main/0.log" Nov 02 02:07:22.077202 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.077171 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lq2pz_305e6b99-4656-459a-95a4-c98da237618f/kube-rbac-proxy-self/0.log" Nov 02 02:07:22.108195 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.108167 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c4dfbbfb-bj6wn_98370bbf-d509-4f16-9ca2-d7f16526eee1/metrics-server/0.log" Nov 02 02:07:22.136239 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.136205 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-p4gjx_bf628013-f478-4707-ab31-c9cd6e5ed0b6/monitoring-plugin/0.log" Nov 02 02:07:22.168790 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.168761 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-msdtj_86828f31-5a23-4d6f-a771-fbf67b94f89b/node-exporter/0.log" Nov 02 02:07:22.195737 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.195709 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-msdtj_86828f31-5a23-4d6f-a771-fbf67b94f89b/kube-rbac-proxy/0.log" Nov 02 02:07:22.226672 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.226648 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-msdtj_86828f31-5a23-4d6f-a771-fbf67b94f89b/init-textfile/0.log" Nov 02 02:07:22.764241 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.764205 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cfvc4_57fd3745-c4ca-4623-9601-48f70b557fc9/prometheus-operator-admission-webhook/0.log" Nov 02 02:07:22.798310 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.798284 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b55d9fb6d-qg8xx_2724697a-5273-4f4a-995c-0515795bea22/telemeter-client/0.log" Nov 02 02:07:22.822469 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.822439 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b55d9fb6d-qg8xx_2724697a-5273-4f4a-995c-0515795bea22/reload/0.log" Nov 02 02:07:22.847616 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.847591 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b55d9fb6d-qg8xx_2724697a-5273-4f4a-995c-0515795bea22/kube-rbac-proxy/0.log" Nov 02 02:07:22.995553 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.995519 4552 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr"] Nov 02 02:07:22.996029 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.996016 4552 cpu_manager.go:414] "containerMap: removing container" podUID="70f36c25-78ef-48bd-aae1-ac4ff42641a2" containerName="revoke-keys" Nov 02 02:07:22.996073 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.996030 4552 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f36c25-78ef-48bd-aae1-ac4ff42641a2" containerName="revoke-keys" Nov 02 02:07:22.996206 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.996136 4552 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f36c25-78ef-48bd-aae1-ac4ff42641a2" containerName="revoke-keys" Nov 02 02:07:22.999644 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:22.999623 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.010481 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.010447 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr"] Nov 02 02:07:23.051622 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.051531 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9w4\" (UniqueName: \"kubernetes.io/projected/60f0c583-456a-49ce-8294-25a41af6f7f8-kube-api-access-xq9w4\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.051622 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.051615 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-proc\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.051824 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.051702 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-lib-modules\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.051824 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.051744 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-sys\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.051892 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.051859 4552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-podres\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.095027 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.094985 4552 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zr67k\"/\"default-dockercfg-89bfr\"" Nov 02 02:07:23.095027 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.094997 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zr67k\"/\"openshift-service-ca.crt\"" Nov 02 02:07:23.095249 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.095017 4552 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zr67k\"/\"kube-root-ca.crt\"" Nov 02 02:07:23.153469 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153438 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-proc\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153478 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-lib-modules\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153502 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-sys\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153554 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-podres\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153579 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-proc\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153593 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-lib-modules\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153586 4552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9w4\" (UniqueName: \"kubernetes.io/projected/60f0c583-456a-49ce-8294-25a41af6f7f8-kube-api-access-xq9w4\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153655 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153648 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-sys\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.153899 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.153658 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60f0c583-456a-49ce-8294-25a41af6f7f8-podres\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.165654 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.165636 4552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9w4\" (UniqueName: \"kubernetes.io/projected/60f0c583-456a-49ce-8294-25a41af6f7f8-kube-api-access-xq9w4\") pod \"perf-node-gather-daemonset-chxvr\" (UID: \"60f0c583-456a-49ce-8294-25a41af6f7f8\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.314682 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.314592 4552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:23.593383 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:23.593350 4552 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr"] Nov 02 02:07:23.594775 ip-10-0-137-1 kubenswrapper[4552]: W1102 02:07:23.594740 4552 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60f0c583_456a_49ce_8294_25a41af6f7f8.slice/crio-83a26bc649d1d2e279d6a6233ea3dc8149bc25585e571ace89b5f70b25ec8ac1 WatchSource:0}: Error finding container 83a26bc649d1d2e279d6a6233ea3dc8149bc25585e571ace89b5f70b25ec8ac1: Status 404 returned error can't find the container with id 83a26bc649d1d2e279d6a6233ea3dc8149bc25585e571ace89b5f70b25ec8ac1 Nov 02 02:07:24.236793 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:24.236755 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" event={"ID":"60f0c583-456a-49ce-8294-25a41af6f7f8","Type":"ContainerStarted","Data":"d2d724403591e5e645b5e27c5f9588d23ea9904ea281148e1e1e20b10dbcb0ee"} Nov 02 02:07:24.237238 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:24.236818 4552 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" event={"ID":"60f0c583-456a-49ce-8294-25a41af6f7f8","Type":"ContainerStarted","Data":"83a26bc649d1d2e279d6a6233ea3dc8149bc25585e571ace89b5f70b25ec8ac1"} Nov 02 02:07:24.237238 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:24.236928 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:24.254959 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:24.254907 4552 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" podStartSLOduration=2.254892859 podStartE2EDuration="2.254892859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:24.25446685 +0000 UTC m=+3144.360255764" watchObservedRunningTime="2026-11-02 02:07:24.254892859 +0000 UTC m=+3144.360681830" Nov 02 02:07:25.320994 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:25.320966 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c8fb7895f-dxl9d_8c5c5331-9226-40e4-80fa-d5a7053376fb/console/0.log" Nov 02 02:07:25.360071 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:25.360040 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-qj4wc_ec0430a9-64d9-40ed-94b3-5ad4c9c826b8/download-server/0.log" Nov 02 02:07:27.144636 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:27.144608 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rd27f_34e4cf42-f58e-442c-857c-1b7c87152cd7/dns/0.log" Nov 02 02:07:27.168287 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:27.168259 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rd27f_34e4cf42-f58e-442c-857c-1b7c87152cd7/kube-rbac-proxy/0.log" Nov 02 02:07:27.305527 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:27.305497 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vts4p_30d617c8-7375-4b58-93b2-7eb39c4bb8c0/dns-node-resolver/0.log" Nov 02 02:07:27.749427 ip-10-0-137-1 kubenswrapper[4552]: E1102 02:07:27.749389 4552 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5cb8t" containerName="extractor" Nov 02 02:07:28.061354 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:28.061282 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zttdn_040c33cc-4366-4e4a-8d93-3f35a1bd5cb8/node-ca/0.log" Nov 02 02:07:29.081545 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:29.081514 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489g8hq9_f1472bbc-66ed-4e4e-830c-87290f40c66a/istio-proxy/0.log" Nov 02 02:07:30.256827 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:30.256797 4552 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-chxvr" Nov 02 02:07:31.312299 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.312271 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q47qx_dd923345-a8ad-45e5-8a8f-ab39df962b65/discovery/0.log" Nov 02 02:07:31.434077 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.434044 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/0.log" Nov 02 02:07:31.435222 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.435204 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/1.log" Nov 02 02:07:31.653033 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.652954 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-p24rm_a36874ab-fccd-45b7-9497-668a0fd1c983/istio-proxy/0.log" Nov 02 02:07:31.685698 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.685666 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-q856k_7f5f593a-2283-4295-acd1-909ae2384f98/istio-proxy/0.log" Nov 02 02:07:31.716559 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.716524 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9_3f3b5284-e92a-42aa-95b0-0fe67d121bd7/istio-proxy/0.log" Nov 02 02:07:31.794928 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:31.794903 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:07:33.762667 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:33.762642 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489g8hq9_f1472bbc-66ed-4e4e-830c-87290f40c66a/istio-proxy/0.log" Nov 02 02:07:34.169794 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:34.169699 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmft6_622df053-819a-4cbb-9ab8-bb507756ac96/serve-healthcheck-canary/0.log" Nov 02 02:07:34.931708 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:34.931679 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5cb8t_0a74bd44-3615-4c98-8f3b-4ba10ff959af/kube-rbac-proxy/0.log" Nov 02 02:07:34.959018 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:34.958985 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5cb8t_0a74bd44-3615-4c98-8f3b-4ba10ff959af/exporter/0.log" Nov 02 02:07:34.987256 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:34.987233 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5cb8t_0a74bd44-3615-4c98-8f3b-4ba10ff959af/extractor/0.log" Nov 02 02:07:36.019713 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.019684 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q47qx_dd923345-a8ad-45e5-8a8f-ab39df962b65/discovery/0.log" Nov 02 02:07:36.088666 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.088633 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/0.log" Nov 02 02:07:36.089898 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.089880 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-4nkxc_a9f2a834-c86c-4af6-bbde-17ba9e356e2d/istio-proxy/1.log" Nov 02 02:07:36.221058 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.221026 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-p24rm_a36874ab-fccd-45b7-9497-668a0fd1c983/istio-proxy/0.log" Nov 02 02:07:36.239488 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.239465 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-q856k_7f5f593a-2283-4295-acd1-909ae2384f98/istio-proxy/0.log" Nov 02 02:07:36.254781 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.254758 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5tkl9_3f3b5284-e92a-42aa-95b0-0fe67d121bd7/istio-proxy/0.log" Nov 02 02:07:36.285839 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:36.285771 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-s25kq_81d8235e-3957-42a3-ae40-7bb53eea23f8/payload-processing/0.log" Nov 02 02:07:37.674636 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:37.674603 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7_60baacdd-b56d-4672-bc18-e3caf6ab4be4/manager/0.log" Nov 02 02:07:37.695352 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:37.695327 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d6cbb5df9-vf8t7_60baacdd-b56d-4672-bc18-e3caf6ab4be4/copy-manifests/0.log" Nov 02 02:07:38.985898 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:38.985870 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-44bc2_9e05de3d-7173-4ca5-ab1e-55872fbe34db/jobset-operator/0.log" Nov 02 02:07:39.324248 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:39.324162 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489g8hq9_f1472bbc-66ed-4e4e-830c-87290f40c66a/istio-proxy/0.log" Nov 02 02:07:39.658133 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:39.658037 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kwxtx_14d4c10d-b4c8-4973-ba7f-56b3e857d4b5/manager/0.log" Nov 02 02:07:39.742864 ip-10-0-137-1 kubenswrapper[4552]: I1102 02:07:39.742822 4552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-v4xbm_5e507870-0bc0-4bb7-b0cb-381084e3a5c1/manager/0.log"